I remember your very interesting interview on Personal Democracy Media. Thank you for sending me the information about your blog. I will be reading and commenting on a regular basis. The URL for my blog digital governing is above. You might find it interesting. My posts generally are more speculative and draw from a broader cross-section of sources than certain others in the e-gov. field.

Michelle

Great summary of what Chicago has done and where it’s going. Looks like an exciting year for people interested in improving cities. Thanks!

Nice link to the music too!

That’s correct, and it’s the strategy we suggest that data publishers use when architecting their dataset. Socrata does maintain our own metadata for when a particular record was created or updated, but for many datasets it isn’t authoritative since it may be some time before the record update finds its way from the system of record to the open data platform. It’s best that the data publisher provide an explicit “Last Updated” field so that its very clear when the update occurred.

As I understand it, that’s the plan for the Crimes dataset and for other datasets where that metadata would be useful.

I’m not totally clear on one thing: you say that update timestamps are part of the metadata in your new process, but also that the Chicago team has been responsive and will take suggestions to heart. I think that means “we’ll have it, but because of our schedule, it wouldn’t be a waste for the city to add it in the meantime.” I guess ultimately the city might be well off including the change data in the dataset anyway, to be completely explicit.

We’re working closely with Chicago’s open data team to improve both our software and the consumption experience for their information assets and resources. We’ve recently deployed a host of upgrades to our platform which have improved both query performance as well as the the speed and frequency with which dataset updates can be performed. We’re working together on new, more tightly-integrated update processes which will allow Chicago to perform in-place updates in a fraction of the time.

One of the Big Data problems that we’re constantly optimizing for is how to keep large datasets of upwards of 5 million rows performant with our caching, full-text indexing, and replication technologies while simultaneously allowing real-time query capabilities for people who want to interactively explore and visualize the data.

We’re also developing the next generation of the Socrata Open Data API (SODA 2.0) which features amongst other things a MUCH more expressive and easier to learn SQL-based query syntax, simpler, cleaner data formats, and tons of other developer-focused features. We’ve gone back to the drawing board and rebuilt the API from scratch to focus on the developer and I’m very excited to be close to sharing that experience with others.

The Chicago team has also been very responsive to feedback about the usability of the data itself – I’m sure they’ll take your suggestion about providing row update timestamps to heart and introduce those into more of their datasets – that metadata is already part of the new update processes we’re working on.

We are proud and fortunate to partner with the City of Chicago and other progressive thought leaders in government to make public data easy to access, share and reuse. Keep the comments coming. We very much value the feedback of the Open Data community.

“Beyond fulfilling a promise from the transition report, why is any of this important? The overarching answer is not about technology at all, but about culture-change. Open data and its analysis are the basis of our permission to interject the following questions into policy debate: How can we quantify the subject-matter underlying a given decision? How can we parse the vital signs of our city to guide our policymaking?”

and

“The City-as-Platform”

“It’s a reconceptualization of government not as provider of end user experience (i.e., the app or service itself), but as the provider of the foundation for others to build upon. Think of an operating system’s relationship to the applications that third-party developers create for it…This model is the hybrid of outsourcing and civic engagement and it leads to better experiences for residents. And what institution needs a better user experience all around than government?”

Posts like this just exacerbate that annoying foundation-staffer tendency to press already-overworked people to find more time to write