postgresql 2 Comments

The People of Postgres: Tom Lane

This post was originally posted on Medium, a new blogging platform made up mostly of people who aren’t necessarily subscribed to Planet. So, please forgive the obvious statements, as the target audience are people who don’t know very much about Postgres. Tom Lane, taken by Oleg Bartunov

Wednesday May 23, with no fanfare, Tom Lane’s move to Salesforce.com was made public on the Postgres developer wiki.

For 15 years, Tom has contributed code to Postgres, an advanced open source relational database that started development around the same time as MySQL but has lagged behind it in adoption amongst web developers. Tom’s move is part of a significant pattern of investment by large corporations in the future of Postgres.

For the past few years, Postgres development has accelerated. Built with developer addons in mind, things like PLV8 and an extensible replication system have held the interest of companies like NTT and captured the imagination of Heroku.

Tom has acted as a tireless sentry for this community. His role for many years, in addition to hacking on the most important core bits, was to defend quality and a “policy of least surprise” when implementing new features.

Development for this community is done primarily on a mailing list. Tom responds to so many contributor discussions that he’s been the top overall poster on those mailing lists since 2000, with over 85k messages.

Really, he’s a cultural touchstone for a community of developers that loves beautiful, correct code.

Someone asked: “What does [Tom’s move] mean for Postgres?”

You probably don’t remember this:

Salesforce.com bases its entire cloud on Oracle database,” Ellison said, “but its database platform offering is PostgreSQL. I find that interesting.

When I read that last October, I was filled with glee, quickly followed by terror. I love my small database community, my friends and my job. What if Oracle shifted its attention to our community and attacked it, directly? So far, that hasn’t happened.

Instead, Salesforce advertised they were hiring “5 new engineers…and 40 to 50 more people next year” for a “huge PostgreSQL project.”

Tom’s move probably won’t change much for the day-to-day operation of Postgres itself. Hopefully, things are about to get real at Salesforce.

I’m a major contributor to Postgres. I started in 2006, learning about relational databases through work at a small bike parts manufacturer and ERP. My contributions include code, starting conferences, encouraging user group leaders and introducing Postgres to communities that otherwise would never hear from us. I’m a data architect at Mozilla.

postgresql

Distributed databases: a series of posts including 2-phase commit in Postgres

There’s a fantastic set of blog posts about distributed databases and network partitioning, starting with this post explaining the perils of trying to “communicate with someone who doesn’t know you’re alive.”

The next post is about Postgres and 2-phase commit. And there are four additional posts in the series.

The whole series worth reading for anyone interested in data stores, consistency and Postgres! 🙂

postgresql 4 Comments

Migrations with Alembic: a lightspeed tour

I’ve got a Beer & Tell to give about alembic. Alembic is a migration tool that works with SQLAlchemy. I’m using it for database migrations with PostgreSQL.

So, here’s what I want to say today:

The most difficult thing to deal with so far are the many User Defined Functions that we use in Socorro. This isn’t something that any migration tools I tested deal well with.

Happy to answer questions! And I’ll see about making a longer talk about this transition soon.

python 5 Comments

TIL: Formatting, search_path and colorcolumn

The last six months have involved a lot more writing of code than the previous couple of years.

I’ve been tweeting little things I learn on a daily basis and thought I’d look back on this week.

format()

A reocurring problem with report writing is getting numbers formatted properly for the occassion. I discovered ‘format’ in Python this week:

print "{0:.2f}%".format(float(1)/3 * 100)

That prints out a float to 2 decimal places. I looked around and Dive Into Python has similar syntax, but without the format() function. So, the equivalent would be:

print "blah %.2f" % (float(1) / 3 * 100)

So, why use one over the other? A user on StackOverflow suggested that compatibility with 2.5 might drive a person to use ‘%’ over ‘format()’, but otherwise, the poster suggested that format() is the cleaner looking and more flexible choice.

set search_path = bixie

I’m working on a new schema for a project. We’re rolling out a prototype quickly, so we’re going to house it in our existing production database for now. To keep things easy to clean up, Laura suggested that we put things into a separate schema. For managing our database models, I’ve switched to using SQLAlchemy, and also alembic for migrations. This made it super easy to specify that I wanted all the Bixie related tables in their own schema:

class BixieCrash(DeclarativeBase):                                              
    __table_args__ = {'schema': 'bixie'}                                        
    __tablename__ = 'crashes'

And that was it.

Then, to avoid having to add ‘bixie.’ to all the table paths in test queries, I put this command into the tests:

 cursor.execute(""" SET search_path TO bixie """)

I imagine there are some other ways to handle this. We’re not really using the ORM for anything other than schema loading, so I’ll probably add that to our connection initialization code for the new app. Then developers can write their queries as without any concerns about being in the correct schema.

And I’ll glow just a little bit about deploying alembic on stage!

set colorcolumn=80

I’ve been trying to write prettier Python. Today’s micro-effort was figuring out how display a vertical line to tell me when I exceed the 80 character width. The proper command to add to .vimrc is:

:set colorcolumn=80

Which looks something like:

colorcolumn in action

speaking , , , 2 Comments

Keynote at OSDC.tw: What Beginners Teach Us

I returned from Taiwan on Monday after a long weekend at OSDC.tw.

I gave a keynote called “What beginners teach us“. Audrey Tang graciously translated the slides for me.

The talk was a bit short – about 30 minutes, which left nearly 15 minutes for questions. So many people asked questions about resources for teaching their families and children Python. My first recommendation was Python Programming: An Introduction to Computer Science. The reason I suggested this was the clearly articulated learning objectives at the start of each chapter.

What we really need, though, is a short list of books oriented toward the different kinds of relationships programmers have (parents, grandparents, colleagues, lovers, children, etc.). Each relationship feels a bit different in terms of what will motivate the person, why the programmer is seeking to educate them and what ultimately will be useful about programming or computational thinking in their lives. I’d appreciate any books or resources you’ve used!

Anyway, the questions were wonderful, and I’m looking forward to the video being posted.

UPDATE: Video is now posted!

WoFOSS

I met members of WoFOSS and chatted about starting a PyLadies Taiwan chapter! WoFOSS has been around at least since 2010, and they have monthly meetings in Taipei. About 70 Taiwanese women are involved, and they are hackers from all kinds of different FOSS communities.

Great hacker talks

I also saw some amazing talks, including one by first-time speaker Andrew Cole, who also publishes a zine in Seattle about pinball. His presentation was a tour of Rosetta Code, which translates bits of code between lots of different languages. He linked to the fabulous “chef” language (not the configuration management tool).

The conference was full of amazing hacks, like this Open Office piano-player (and many more audio/pictograph hacks) from imacat. This hack used OO Calc to create a playable piano, and another sheet plays a Christmas tune that reveals an image of Santa flying reindeer.

More fun hacks came from Yusuke Kawasaki. His talk was a very funny set of demos using iPhones and a very simple IR transmitter, made from extremely cheap parts and Sguru. He later demo’d a two-iPhone camera and remote control system for a remote control dump trunk at the hackathon.

On day two, Audrey Tang gave an amazing demo of making an open source dictionary using Postgres and Node out of documents available only in Excel from the government. I was inspired.

I highly recommend OSDC.tw, which typically happens in April every year.

advice 1 Comment

Tech literacy and learning to code for girls in middle and high school in Portland, OR

A friend asked about programs suitable for a 10 year old and a 14 year old girl in the Portland area.

Here’s what I came up with:

As far as things that are already underway: http://www.chicktech.org/ has the most stuff for the 14-year-old

Looks like it happened in January: http://www.chicktech.org/participants/workshops/

Next there’s FreeGeek: http://www.freegeek.org/

http://www.freegeek.org/volunteer/

Their “adoption” program is interesting because you learn how to put together your own computer. Highly recommended.

There’s online programming courses offered through ORVED: http://www.orved.org/

What did I miss?

From twitter:

postgresql

PostgreSQL security releases now available: versions 9.2.4, 9.1.9, 9.0.13 and 8.4.17

PostgreSQL Global Development Group has just released updates for all currently supported versions of PostgreSQL.

From the release announcement:

The PostgreSQL Global Development Group has released a security update to all current versions of the PostgreSQL database system, including versions 9.2.4, 9.1.9, 9.0.13, and 8.4.17. This update fixes a high-exposure security vulnerability in versions 9.0 and later. All users of the affected versions are strongly urged to apply the update immediately.

A major security issue fixed in this release, CVE-2013-1899, makes it possible for a connection request containing a database name that begins with “-” to be crafted that can damage or destroy files within a server’s data directory. Anyone with access to the port the PostgreSQL server listens on can initiate this request. This issue was discovered by Mitsumasa Kondo and Kyotaro Horiguchi of NTT Open Source Software Center.

I wanted to highlight a couple things from the FAQ we developed for this release.

  1. There are no known exploits for the major security issue fixed by this release. The vulnerability was discovered through security testing conducted by NTT.
  2. Only users of 9.0 PostgreSQL and higher are affected by the major vulnerability.
  3. Affected users are those who allow unrestricted access to the network port PostgreSQL listens on. If you allow anyone, without IP address whitelisting, firewalling or some other kind of network-based access control, to connect to your network port, you are especially vulnerable.

Upgrading from minor version (9.2.3 to 9.2.4, for example) only requires that you install the new binaries and then restart PostgreSQL.

Additionally, if you are using GiST indexes, read the detailed notes in the release announcement to see if you are using features that require you to REINDEX your GiST indexes.

Please update as soon as possible!

Many thanks to our volunteer packagers who worked hard for the past several weeks to make this release possible. All PostgreSQL software releases are managed by volunteers.

speaking, teaching , 12 Comments

About high school computer science teachers

I’m giving a talk at PyCon next Saturday about teachers. The title is “What teachers really need from us“.

The first thing I should admit is that when I started thinking about this talk, I was sure that the list of what teachers needed from us was really long.

Then, I started actually talking with teachers.

So, here’s what some of them have said:

  • Reading comprehension is the biggest barrier to completion of AP Computer Science (Page 8 of this AP CS course description)
  • Fighting for continued existence is the biggest battle for a computer science teacher every year. “The number of secondary schools offering introductory computer science courses dropped 17 percent from 2005 to 2009 and the number offering Advanced Placement (AP) computer science courses dropped 35 percent in that time period.” December 2010 report
  • Writing personal letters from a teacher to students and parents increased the number of girls in one teacher’s class (in Virginia) from nearly zero to 50%. Research into increasing the number of women and minorities in CS classrooms is available in Stuck in the Shallow End.
  • Students at a high school learned three languages in three years. (C++, Java and Python) This busted so many notions I had about how long learning to program takes or what languages are most appropriate for beginners.
  • Kids don’t need algebra to learn to program. Algebra is a weeder course, often a prerequisite to CS and one that strongly indicates whether or not a student will graduate high school. What if kids could take an “algebra on computers” course instead of failing out of school? Please note, learning to program is not the same thing as being a professional programmer.
  • School counselors who help kids choose classes still send students to CS class believing that they’re going there to learn to type. Find out more about the wildly varying understanding state-by-state of what a computer science class really is in the Running on Empty report.
  • What teachers wanted from me was for me to come to their classes and give a short talk to their kids about myself and my work.
  • Teachers were super excited to hear about PyLadies. They struggle to get girls into their classes and are all looking for ways to increase the diversity of their classes.
  • The CS teachers I’ve met want to share their lessons – with me and with other teachers.
  • The CS teachers I’ve met don’t know other CS teachers.
  • Teachers were only mentioned once in the 84 initial statements of support for code.org

I think we’re all really missing out when we don’t talk to teachers.

I’ve talked directly with nine computer science teachers. Most of them are in Oregon, but I also was introduced to a couple teachers who came to Python-related conferences, or were married to Python programmers. I’m hoping to meet more. If you know someone, please put them in touch with me. I’m happy to chat over the phone or email, and love to meet folks in person.

feminism, personal, user groups , 1 Comment

Why PyLadies?

pyladies_blue

Hey Hacker News! If you’re coming here for the first time, you may also be interested in What I mean when I say I would like more women in the software industry

PyLadies is a group of women working on welcoming, encouraging and directly inviting women to join the Python community and to learn from each other.

I started a PyLadies chapter in Portland, OR last September (2012). We started out with weekly meetings to do homework from a Coursera class to make games with Python. That turned into weekly meetings — plus homework meetups on Saturdays at a local coffee shop, and IRC hangout time to test homework. And that turned into me giving mini-lessons at each Coursera meetup about the material from the class.

People seemed really excited.

Stats - PyLadies PDX (Portland, OR) - Meetup

Before we knew it, it was December, we had over 60 women subscribed to the Meetup, 30 of which had attended a meeting. Today, we’ve got 96 subscribers, 50 people have attended a meeting, and more have signed up to attend events in the future than ever before. And, it’s done by women. Using open source. Teaching classes. Learning developer tools. And writing software.

Since September, I’ve met even more women involved in running PyLadies chapters across the country. Much like the way the PostgreSQL community is organized, we’ve got a loosely connected group of people working independently. We offer support to each other, but don’t have hard and fast rules about what each chapter does. We encourage teaching and workshops, but don’t require them. We share our resources and are quick to put git repos out there of our materials. We send lots of pull requests. And we’re constantly looking for ways for women to get more involved in open source and Python.

All Group Reviews - PyLadies PDX (Portland, OR) - Meetup

I’m completely energized by the positive feedback we’ve gotten for every meeting. More recently, I’ve heard from people that they feel confident and sure of their knowledge because they’ve spent time in our meetups talking and learning from other women.

My goal is to make every get together like that – by having great lessons, a shared understanding of coaching and peer-based education and presentations from our members. Building these groups takes time, and I’m impatient to get to the part where I feel like every interaction with the group is rewarding for every member.

And I can’t do it alone. We’ve got four meetup organizers (although one is about to relocate to the Bay Area!). I work closely with Flora Worley, a kickass developer who chose programming as a career path after working on a PhD, on topic details and planning for the meetings. I’m so looking forward to meeting in person with the many members of the PyLadies community at PyCon next month.