The Origin Story of Django: From Local Newspaper CMS to Global Framework and Beyond

Next Day Video

Summary:

Simon Willison recounts the origin story of the Django web framework, starting from his internship at the Lawrence Journal-World in 2003, and highlights various projects he built using it.

• Django was initially developed as an internal Content Management System (CMS) to rebuild Lawrence.com from PHP to Python, driven by Rob Curley's philosophy of rapid "build cool shit" development.

  

• Early projects included the 6 News Lawrence weather page and the complete Lawrence.com rebuild, showcasing Django's ability to quickly build complex applications.

  

  

• They brainstormed several names for the open-source release, including "The Tornado Publishing System" before settling on Django.

  

• The "wouldn't it be cool if" philosophy drove features like MP3 downloads of local bands linked to their gig schedules, leveraging relational database technology.

  

• At Yahoo, he built Yahoo AstroNewsology, a mashup of news and horoscopes, for an internal hack day, demonstrating Django's versatility beyond news sites.

  

• At The Guardian, he delved into data journalism, creating tools like a heat map of British National Party members using leaked data and a crowdsourcing platform for the UK MP expenses scandal.

  

  

• He also developed personal side projects, including Django People (a community map) and Wildlife Near You (a crowdsourced animal sighting database), leading to the mobile-focused Owls Near You.

  

  

  

  

• A collaborative hackathon at a Napoleonic sea fortress led to Bugle, a "Twitter-like" application for offline collaboration, predating Slack.

  

• His startup, Lanyard, an event discovery platform integrated with Twitter, eventually led to his role at Eventbrite, where he developed the production-focused Tiki Bar debugging tool.

  

  

• Throughout his career, Django enabled him to consistently "build cool stuff" and ship it efficiently, a philosophy learned from his early days at Lawrence Journal-World.

Introduction and Talk Reorientation [00:00:00]

The speaker, Simon Willison, reorients his talk to focus on the origin story of Django, acknowledging that it's the 10th birthday of the framework and its genesis hasn't been widely shared. He intends to cover Django's origins first, followed by a selection of projects he has built with it.

The Genesis of Django at Lawrence Journal-World [00:00:56]

Simon Willison's journey with Django began in 2003 through a web programmer job opportunity at the Lawrence Journal-World.

• Hiring and Tech Transition [00:01:00]
  • Simon, then a university student in England, responded to a job ad for a PHP/MySQL developer position in Lawrence, Kansas.

    

  • Upon arrival, he and Adrian Holovaty, the original developer, decided to switch from PHP to Python, influenced by Mark Pilgrim's "Dive Into Python," due to PHP's limitations for large, maintainable systems.
  • They both had strong opinions on web development, including URL design, HTTP methods, and using CSS, which existing Python frameworks didn't fully support.
• Lawrence Journal-World's Unique Environment [00:03:07]
  • The Lawrence Journal-World was a small newspaper with a circulation of about 10,000 but invested heavily in its newsroom and software development.
  • Rob Curley, the boss, fostered an unofficial mission statement: "build cool shit," encouraging rapid development and ambitious projects.

    

  • Lawrence.com, initially built in PHP by Adrian, was a highly comprehensive local entertainment website, showcasing the type of complex system they aimed to rebuild with Python.
    • The original Lawrence.com was a very clean PHP codebase but was reaching its limits.
• First Django Projects [00:04:47]
  • The initial goal was to rebuild Lawrence.com using Python, which they internally referred to as "the CMS" before it was named Django.
  • The first website shipped with Django was the 6 News Lawrence weather page, featuring a dynamic illustration of the Lawrence skyline that changed based on weather conditions.

    

  • The full 6NewsLawrence.com website followed, with its launch comically delayed because TV personalities disliked their headshots.
  • By the end of Simon's internship, Adrian had rewritten all of Lawrence.com using the new Python CMS.

    

    • Django was built up to recreate Lawrence.com, in contrast to Rails being extracted from Basecamp, meaning Django was iteratively developed to meet known requirements.
• The "Wouldn't it be cool if..." Philosophy [00:06:50]
  • This philosophy replaced formal user stories and specs, promoting rapid development and shipping cool ideas immediately.
  • A prime example was a little league sports site (LWWorld.com Game 2008), conceived and built in just three days, featuring player profiles, schedules, photos, results, SMS notifications, and 360-degree interactive field photos.
    • Django, even at its early stage, provided the necessary primitives to build such complex interactives quickly.
  • Another instance was adding MP3 downloads of local bands to Lawrence.com, linked to their gig schedules. This demonstrated the power of relational database technology for creating interconnected features.

    

    • Simon notes that this "ship it" philosophy, while rewarding, sometimes leads to long-term maintenance burdens for features that seemed like a good idea at the time.
  • The early names brainstormed for the open-source release of "the CMS" included "Brazos," "Webbing," "Physique," "Anson," "The Tornado Publishing System" (Simon's favorite, for "TPS reports"), "Private Dancer," "Fizgig," "Lavalier," and "Pythy."

    

Django Projects at Yahoo [00:10:58]

After his internship, Simon joined Yahoo UK, working for an R&D team in the States.

• Yahoo R&D Experience [00:11:12]
  • He learned that R&D teams often struggle to ship products, spending a year and a half primarily on prototypes.
  • One project, Yahoo Fire Eagle, eventually became a real product after he left.
• Yahoo AstroNewsology [00:11:40]
  • For an internal Yahoo hack day in 2005, Simon built Yahoo AstroNewsology using Django, an application that combined Yahoo News feeds with Yahoo Horoscopes.
  • It pulled celebrity names from news, found their birth dates, looked up their horoscopes, and presented them on a single page.
  • A humorous coincidence occurred when Dick Cheney's horoscope perfectly aligned with news of his hunting accident during the week the project was built.

    

Django in Data Journalism at The Guardian [00:13:51]

After a period of consulting, Simon joined The Guardian, drawn by the newsroom environment and the emerging field of data journalism.

• Joining The Guardian [00:14:08]
  • He missed working in a newsroom and was inspired by data journalism work at other major newspapers.
  • The Guardian wanted to explore faster development cycles to integrate with the news, as their existing Java CMS was slow.
• Discovering Simon Rogers' Data [00:15:20]
  • Through staff onboarding, Simon met Simon Rogers, a journalist whose specialty was gathering data for infographics.
  • Rogers had hundreds of meticulously researched, news-quality Excel spreadsheets on his hard drive, containing data on "everything you could imagine."
  • This discovery led to a collaboration where Simon (Willison) would use the data provided by Simon Rogers for web projects.
• BNP Membership Heat Map [00:17:03]
  • One of the first data journalism projects involved analyzing a Wikileaks dump of British National Party (BNP) member names and addresses.
  • To avoid privacy violations, they geocoded the addresses to parliamentary constituencies and generated a choropleth map (heat map) of the UK, showing hotspots of BNP activity.

    

  • The map was produced in SVG, allowing it to be easily used online and printed in the physical newspaper the next day, demonstrating rapid cross-platform deployment.
  • Simon jokes that the self-service map-building tool he created for journalists ran on his Linux desktop under his desk for six months after he left, eventually being converted into a persistent VMware instance.
• Crowdsourcing MP Expenses Scandal [00:20:11]
  • His favorite project at The Guardian addressed a scandal where UK Members of Parliament (MPs) were caught fiddling expenses.
  • After a rival newspaper leaked the full expense reports, The Guardian launched a crowdsourcing website in five days to analyze 450,000 pages of documents.

    

  • Users could input their postcode to see their local MP's expenses and categorize pages as "not interesting," "interesting," or "investigate further."
  • The community analyzed hundreds of thousands of pages within 18 hours, uncovering newsworthy stories and engaging the public.
• The Data Blog [00:23:20]
  • To make Simon Rogers' extensive spreadsheets publicly accessible, they started "the data blog," publishing the raw data as Google Spreadsheets.
  • This fostered a community of "data nerds" who created their own visualizations and extracted additional value from the published data.

Personal Side Projects with Django [00:24:15]

Simon began building side projects with his future wife, Natalie, leveraging their complementary skills (her frontend, his backend and ops).

• Django People [00:24:44]
  • Their first project together was Django People, a map allowing Django community members to mark their locations.

    

  • It still exists today, hosted by Django Project, after persistent community requests to keep the code and data alive.
• Dev/fort and Wildlife Near You [00:25:31]
  • A collaborative hackathon, "/dev/fort," involved renting a Napoleonic sea fortress in the Channel Islands for a week of building.
  • The resulting project, "Wildlife Near You," aimed to answer "where is my nearest llama?" by crowdsourcing animal sightings.
  • Users could create trip reports, import photos, and build profiles of places and animals seen.

    

  • The platform could then search for animals near a given location, e.g., finding the nearest llama.

    

• Owls Near You [00:28:27]
  • Extending "Wildlife Near You," they built "Owls Near You," a mobile-first application using HTML5 geolocation to instantly tell users their nearest owl (or other animals like lions and monkeys).

    

  • It was noted for its simplicity as it didn't require user accounts or ongoing moderation, a lesson learned from the complexities of "Wildlife Near You."
• Bugle [00:30:02]
  • Another project from a dev/fort was Bugle, a "Twitter-like application for groups of hackers collaborating in a castle... with no internet connection."
  • It served as an internal communication tool with features like a pastebin and to-do lists, predating Slack.
  • A "Magic Twitter support" feature even allowed Twitter apps on phones to connect to Bugle via local DNS redirection.

    

Lanyard and Eventbrite [00:31:30]

Simon's most recent project discussed was Lanyard, which evolved into a startup.

• Lanyard [00:31:43]
  • Lanyard was a "honeymoon project" built with Natalie during a two-week stop in Morocco after contracting food poisoning during their travels.
  • It was designed to help Twitter users find conferences and events.

    

  • The project quickly went viral, leading them to apply for Y Combinator and spend three years building a startup.
  • Simon cautions that while startups seem "amazing" from the outside, they often involve "misery, pain, anguish, and stress."
• Eventbrite and Tiki Bar [00:33:42]
  • Lanyard was eventually sold to Eventbrite, leading Simon to San Francisco.
  • At Eventbrite, he developed "The Tiki Bar" at a hack day, a production-focused debugging toolbar for Django.
  • Unlike the standard Django debug toolbar, Tiki Bar is designed to run in production with minimal overhead, providing detailed timelines of SQL queries and service calls for difficult-to-debug issues.

    

  • A unique feature is that the eyes of a tiki god icon glow red in disapproval if a page takes over 500 milliseconds to load.

Conclusion [00:34:52]

Simon concludes by reflecting that his career highlights with Django underscore the enduring influence of Rob Curley's "build cool shit" philosophy. He emphasizes that Django makes it easy to build and ship innovative projects.