This time, I was better prepared. I knew what to expect and leveraged my time better, all while having a ton of fun.
This was my second time attending an in-person PyCon US. My first one was in 2019. I remember being awestruck by the ocean of people. I was glad that I had already attended DjangoCon US and some regional conferences. I recognized some people in the crowd, and it made it feel like a smaller event.
I am thankful for and proud of my friend Mariatta (PyCon US Chair) who oversaw it.
PyCon US 2024 was truly incredible. If you ever have the chance to attend, I highly recommend it.
Disclaimer: the content of this post is a reflection of my career journey and not specific to my work at JPMorgan Chase & Co.
After a busy day, I was tempted to get some cheap takeaway for supper, but I thought twice, and decided I should enjoy the best this city has to offer. So, I googled “best restaurant in Pittsburgh.” The top result was Fig & Ash , a short walk away. I went there and had an incredible meal.
Standing on a bridge on the Allegheny River with a view of the downtown and one of the many other bridges
This was my first time in Pittsburgh, and I didn’t know what to expect. Downtown Pittsburgh, with its many steel bridges and brick buildings surrounded by riverfront, was a surprisingly gorgeous scene.
I arrived on Wednesday, settled in, and immediately took some time to see the David L. Lawrence Convention Center and the downtown.
At the Coiled booth, I learned about using Dask for parallel computing. Dask was a projects cited in the Measuring the performance of CPython.
At the PostgreSQL community booth, I obtained some learning resources, including a paper version of PostgreSQL FAQs .
As a fellow Lawrence native, Elizabeth seems to be hoping for a Lawrence Django conference as much as I am. :) I missed the Lawrence Technology Conference this year, but hope to make it next year at least, if there is one.
At the Crunch Data booth, I met Elizabeth Garrett Christensen. I know of Elizabeth through the talks she has given at DjangoCon US: “ How to Be a Postgres DBA in a Pinch ” and “ Postgres Performance: From Slow to Pro ”. Crunch Data provides Enterprise PostgreSQL support, including fully managed Postgres as a Service.
At Snowflake booth, I learned about Snowflake Quickstarts and using Snowflake tasks for orchestration.
At MongoDB booth, I caught up with my friend Mark Smith (MongoDB) and obtained some learning resources from Rita Rodrigues (Director of Developer Relations at MongoDB).
At the Oracle booth, Gary Brenner (Oracle Senior Cloud Engineer) told me about Oracle Cloud and self-managing Oracle databases.
At the Edge DB booth, Yury Selivanov (Edge DB Co-founder) told me that Django will be supported in a few months and showed me the impressive performance improvement created by Edge DB. Looking forward to learning more!
At Datadog booth, Julian Levi Hernandez (Datadog, formerly CockroachDB) gave me a demo of a dashboard using Cockroach audit logs to further drill down into metrics.
In the Expo hall, I did a tour of database companies and other fun tools.
The night finished with a bear hug from Carol Willing (CPython Core Developer, Python Steering Committee) who told me I am awesome. :)
It was so good to see my friend Jess Garson (Elastic, formerly Twitter) at the Elastic booth and meet Philip Krenn (Elastic). Jess and I first met at PyGotham 2019, and she introduced me to many aspects of the Twitter Developer Program. :)
I got to hang out with Meagan Voss (Wagtail) and Nic James (DjangoCon US 2019 Chair) and talk about our work.
It was a pleasure to chat with Simon Willison (Django Co-founder) again briefly before he left to work on his keynote. I knew he would nail it.
After years of following each other on social media, it was wonderful to meet Anthony Shaw (Microsoft) in person. Along with Iqbal Abdullah (LaLoka Labs, PyCon JP), we talked about the CPython Internals book Anthony wrote and my desire to understand CPython better. Anthony later suggested that I submit to the CfP next year, and I have to admit that it’s tempting.
I gave my first conference talk at the first conference I ever attended. It was DjangoCon US 2017 in Spokane, Washington. I distinctly remember that Roger Masse (United States Senate) and Mark Lavin (Nvidia) attended my talk. At the PyCon US 2024 opening reception, while catching up with Jon Banafato (PyGotham Chair), I saw Mark walking by and couldn’t resist stopping him to introduce myself and thank him years later for going to my first talk. :)
View from the station at the bottom
I had all of my travel plans in place to attend PyCon US 2020 in Pittsburgh and planned to ride the Duquesne Incline , but the in-person conference was canceled due to Covid. I finally had the chance to ride it!
When I heard that the Heinz History Center has the Mister Rogers set, I had to go.
On the way to the Andy Warhol Museum , I saw my friend Andy Knight (Automation Panda), and we walked there together. Andy had given a tutorial “ def test_crash_course_with_pytest(): ”.
Although they were hard at work, I also got to see Deb Nicholson (Python Software Foundation Executive Director) for the first time again since DjangoCon US 2023, Ee Durbin (PSF Director of Infrastructure), Christopher Ngeuebauer (PSF Board Member), and Mariatta. :)
It was good to chat with Mason Eggers (PyTexas Chair), Kevin Horn (Dallas Fortworth Pythoneers Co-founder and PyTexas Organizer), Josh Schneider (PyTexas Organizer), Andy Fundinger (Bloomberg), and Heather Crawford (Bloomberg) again so soon after seeing them at the wonderful PyTexas conference I attended in April. Andy and I first met at PyGotham, another one of my favorite conferences.
As usual, Trey had an interesting gaggle of people around him that included Marie Roald and Yngve Mardal Moe ( Turtle Thread ) and Rodrigo Girão Serrão ( Python Math Teacher ).
I was thrilled to catch up with some of my DjangoCon US friends: Afi Gbagado (DjangoCon US), Abigail Mesrenyame Dogbe (Technology Chief of Staff and Open Source Founder), Noah Alorwu (Marygold and Co.), Felipe de Morais (AfroPython), Jeff Triplett (RevSys, DEFNA Board Member), Eric Matthes (Author of Python Crash Course ), and Trey Hunner (Python Trainer, Python Morsels). I made a new friend Amanda Viera (AfroPython).
Once inside, I soon saw Guido van Rossum (Python Creator and BDFL) walking nearby. He sat down next to me, and I had the chance to chat with him about life and work. I thanked him for spending the time, and he thanked me. I’ve had the chance to spend time with him at three conferences now, and it’s always fun. The Python community is lucky to have a BDFL who actively engages with us and cares.
I couldn’t resist stopping at the convention center to see what was going on.
As I was walking down the riverfront, I saw a man taking photos. He looked familiar. I was pretty sure we had been following each other on social media for a while. I couldn’t resist approaching him. It was Peter Wang (Anaconda Co-founder). It was really fun to meet him, and I found out that he and my friend Andy Fundinger had been consultants at JPMorgan Chase years ago.
I took the day off to sightsee.
After my incredible meal on Thursday night at Fig and Ash, I wanted to do something simple for supper. Based on a tip from another conference attendee, I ended up across the street from the convention center at Condado Tacos . The food was pretty good, but the best part was sitting and chatting with Bloomberg Python Trainers Heather Crawford and Scott Irwin. I met Scott at PyGotham 2018 where I attended a talk he gave “ Dataclasses are here. Now what? ”. I met Heather at PyTexas 2024 where she did an awesome talk “ Python Code Versus Pythonic Code: What Experienced Developers Find Challenging About Learning Python ”, which she also gave at PyCon US.
Abdur-Rahmaan Janhangeer (Python Usergroup of Mauritius (PyMUG)) and I have been following each other on social media for quite some time, but I didn’t know what he looked like. Fortunately, he found me and gave me a wonderful gift of Chartreuse tea, the best selling tea in Mauritius. Thank you!
Earlier this year, I created a page of favorite conference snapshots . Benedict Kofi Amofah (Python Ghana, DjangoCon US) tweeted to me that he’d be in the gallery soon. I took him up on it. :)
It was also a pleasure to meet Brianne Wilhelmi (Springboard) and Aaron Clark (Hamilton Beach) at lunch, and see Ashia Zawaduk (Muck Rack) again after PyCascades 2020.
I found out that Julia Ferraioli (AWS Open Source) was at the conference. I was ecstatic and hurried over to meet her. I had hoped to meet Julia in person for years. In March of 2021, I reached the final round of a Twitter Open Source Program Manager interview process. I was disappointed when I was not offered the job, but later found out that I had been considered alongside Julie and was honored. In a twist of fate, in August of 2022, Julia kindly met with me via Zoom and gave me excellent career advice.
I had the chance to say hello to Sarah Kuchinsky (PyCon US Organizer) for the first time since PyCon US 2019. She was recently the technical reviewer for Al Sweigart’s great book about recursion called “ The Recursive Book of Recursion ”. I really enjoyed it!
It was great to see Andrew Godwin (Django Async Architecture) and Dave Forgac (PyOhio) again.
I saw Pablo and Barry together at the PyCon US 2024 Opening Reception. It was wonderful to be able to share with them five years later how my career has taken off.
At PyCon US 2019, Pablo Galindo Salgado (CPython Release Manager, Bloomberg) was one of the first people I met as a wide-eyed newcomer. It was the Thursday night before the conference talks began, and I was at the conference hall for the swag stuffing, which is a great place to meet people. Scientists had recently photographed a black hole for the first time. Pablo, a physicist, invited me to a table nearby where he demoed some Python astronomy software for me. I later met Barry Warsaw (Python Steering Committee, Nvidia) at the sprints and talked to him about my career. At the time, I was a part-time open source maintainer of Pinax.
Things you can do to help
Black Python devs had a goal of sponsoring seven local conferences in Black communities and hoped to raise $5,000 at PyCon US. They raised over $15,300 from 282 donations.
“Say it loud.” Everything they do is out in the open, proudly showing up for each other, saying, we’ve got this, and celebrating.
Supporting a Django Girls event that gave not just Black women devs, but Black girls a wonderful first interaction with the Python community
Convening leaders from across the globe at conference lunchtime to help understand the situation happening in Africa and what could be done to assist
Converting community sponsorship into tickets for first time attendees who are Black Python Devs members
Facilitating four leaders traveling to their first PyCon US
The Python community can extend the reach further and Black Python Devs can help change perceptions in communities.
The leadership model is “diversity by inclusion.” Global leaders are represented at the table: from many countries, speaking different languages, 50/50 gender parity. It’s members are people who are already around you in PSF, DFS, Djangonauts, Pylades, DjangoGirls. Because the community came from Africa, its leaders do too.
They are now officially a non-profit powered by the GNOME Foundation.
Five members launched a discord. There are 427 members now. Their only plan was to be there for each other.
Jay started working behind the scenes to get as many Black people as possible to PyCon US 2023. There were 3x more than in 2022. They were able to be in the same room together and talk about the experiences they have as Black Python devs that others can’t comprehend, not about how to increase diversity.
Jay listened to people in the community who had experience running non-profits, starting communities, running conferences: Marlene, Dawn, Kojo. He needed to make the community safe, equitable, accessible. This goes beyond a code of conduct to issues that Black people think about everyday like physical safety in public. He also wants to prevent burnout.
Jay decided to create something that would serve as a Wakanda for Black Python devs. Something that would be worth staying for. It’s called Black Python Devs .
Jay asked, how do we get more black people to experience the PyCon community, build friendship, and become repeat attendees?
Many people go to conferences to hang out with their friends. If you don’t have that kind of experience, there’s no reason to attend.
Jay was happy to not be the first Black person keynoting PyCon US. But, at PyCon US 2022, Jay counted only 12 Black people among thousands. He talked to them. They had to have a reason to be there. He thought of the keynoters who came before him. They came, they spoke, they left. He wondered if “the juice wasn’t worth the squeeze.”
Jay talked about beautiful moments from DjangoCon US 2023 in Durham, North Carolina: Black leaders, from all around the world, getting on stage, getting in front of their booths. They showed up and showed out for who they were.
Jay knows two staff developer advocates who are Black and he is one of them. When he sees another Black developer on stage, it’s not just another Pythonista.
At breakfast, I met Loren Crary (PSF) for the first time after following each other on social media for quite some time and saw Al Sweigart (Author of Automate the Boring Stuff with Python ), one of my favorite authors and presenters, again. I also finally met Tania Allard (Quansight, formerly Microsoft) and learned more about her work, including consulting at JPMorgan Chase. It was a pleasure to meet Daniel Graham (SauceLabs) and Jing Cao (National Microbiome Data Collaborative), too.
Breakfast¶ Koushik Krishnan (Microsoft Azure Cosmos DB Engineer) gave his talk “Rest Easy with Jupyrest: Deploy notebooks as web services” at PyTexas. When I saw during the talk that he works on Microsoft Azure Cosmo DB, I was keen to meet him and did. He gave the talk at PyCon US too. It was a pleasure to have the chance to chat with him about different databases over breakfast. I also had the chance to promote Bar Camp Philly, organized by Tim Allen and now Dawn Wages’ wife. ???? back to top
Simon Willison Keynote¶ Simon talked about Large Language Models such as Chat GPT, Google Gemini, Claude, and Llama. He does not think of them as Artificial Intelligence. He thinks of them as Imitation Intelligence. They predict the next word in a sentence. When they get good at that, it’s spooky what they can do. He acknowledged that they are flawed, but just because a tool is flawed, doesn’t mean it’s not useful. When evaluating a new technology, Simon asks, what can I build with this that I couldn’t have built with it before? LLMs open up new options unlike anything he has ever seen. How can we tell which of these models works best? Vibes. Simon recommends using LMSYS Org Chatbot Arena. You vote on which model gave the best response to a prompt, then they are scored using Elo rating system. Simon is relieved, because “openly” licensed models are beginning to rank. LLMs can be run on smartphones now, even with no internet connection. An app called MLC Chat will give you access to Mistral, one of the best openly licensed models. Simon has deep respect for what is known as prompt engineering. It’s not easy to get LLMs to do what you want them to do. Building something simple is easy. Building something production ready can take months and is much harder than people expect. A few tricks Chat prompting
Retrieval Augmented Generate (RAG)
Function calling (“tools”) in a loop Simon coined the term “Prompt Injection” after SQL Injection. He believes that we are not seeing digital assistants, because no one knows how to build them securely. 99% effective isn’t good enough. Never mix untrusted text with access to tools. Simon showed how he gave the LLM a GeoJSON file with different line segments and prompted it to turn it into a single polygon. This “side project” took 3 1/2 minutes to complete Simon showed how he had created a counter that incremented each time he said AI or artificial intelligence. It took 6 minutes to get to prototype and 20 minutes to get it polished. Simon walked through how he passed structured data from the PSF Board Resolutions page into an LLM to parse it into a structured data table that could be filtered. He used a plugin he had been developing called llm.datasette.io. Simon has been able to use LLM to more quickly build things that he couldn’t justify spending the time to build by hand. Tips Ask it to give options (it’s more likely to give a better answer)
It will rarely get the answer right. Ask it to “do better.” LLMs make mistakes. It’s up to you to verify. The tool gets you 90% of the way there. Rather than Generative AI, Simon likes to think of these tools as Transformative AI. Personal AI ethics: Simon came across the term “slop” and likes it. Similar to spam, ask yourself, am I creating unwanted junk or am I using these tools in a responsible way? Simon pondered if using LLMs is cheating. We care if students cheat, because it hurts them and they have an unfair advantage. LLMs feel different. Simon’s whole career has been about getting things done more quickly, including by using open source code. A few rules of thumb Never commit any code that you couldn’t actively explain to someone else
LLMs are good at explaining code. Give it code in a language you don’t understand and it will explain it with 90% accuracy
Then, help people understand how you did it! He said software engineers are uniquely positioned to take advantage of LLMs, because LLMs are good at generating code, and you can check the code by running it. Simon is optimistic. You need to have a CS degree or spend a lot of time learning to do the simplest of computer tasks. LLMs opens programming up to a much wider community. Simon believes that we have a responsibility to not leave anyone behind. ???? back to top
Hallway Track¶ It was wonderful to see Moshe Zadka at PyTexas where he gave a talk “Iterate, iterate, iterate” and again at PyCon US where he gave me a Moleskin journal from Anthropic for recording convo starters for next-generation AI assistant Claude. Moleskin from Moshe At DjangoCon US 2023, Marc Gibbons gave a wonderful talk called “Empathetic testing: Developing with compassion and humility”. Not only was it his first conference talk, but after 44 chemo treatments, 15 fractions of radiation, and a stem cell transplant to treat Hodgkin Lymphoma, he had been given a clean bill of health the week before. It was wonderful to have the chance to catch up with him at PyCon US. ???? back to top
Overcoming GIL with subinterpreters and immutability¶ Yury Selivanov showed a screen with a single Python process running 10 async IO event loops saturating the 10 CPU cores on his laptop, all sharing memory and exchanging values between them from 1 million keys and values and done fairly efficiently. Yury said it is possible to build fast things with Python. Yury kind of likes the GIL. When he tries to do free-threading or multi-thread programming, bad things happen. Free-threading will take a couple of years to be stable. Python subinterpreters are relatively new. You can run these subinterpreters in the same process, side-by-side, isolated from each other, each with its own GIL and occupying one CPU core, sharing the same memory space. One of the subinterpreters will be the main one and will spawn workers and perhaps can have queues between workers and spawn tasks. What if we could safety share state between the subinterpreters with the potential for lots of it and without using pickle, because it’s slow? Safe shared state… no pickle Yury has created a minimal library called memhive that enables efficient data sharing between isolated subinterpreters. He showed a code snippet. Architecture Level 1: simple protocol that enables you to send Python data across subinterpreter boundary
Level 2: higher level API (queues and sychronization primitives)
Level 3: async IO bridge Create memhive and define the async workers that will be running in each separate subinterpreter, each with its own async IO loop. They can access shared state, listen to messages, send messages, spawn worker subinterpreters. The main interpreter will initialize the shared state, broadcast signals to all subinterpreters, push work so that one starts working on a task, listen to messages. Rather than how it works, Yury felt it was more important to talk about how it’s implemented. 3 things to talk about Immutability
Efficient immutability
How we can benefit from it Immutability Python has immutable types: str, int, floates, bytes, tuples
You cannot change an existing object. You have to create a new object out of the existing object.
There is no immutable dictionary or mapping in CPython, which Yury says is a shame. Efficient immutability Creating a new tuple out of an existing one is efficient (O(n)), but you are not usually dealing with a lot of records.
The same inefficiency would be very bad for a dictionary, because we are potentially putting millions of items in them.
Dictionaries can be used as caches, but rebuilding it every time would be unacceptably slow.
Fortunately smart people created a nice algorithm that can implement this with O(log n) The trick Yury would show used trees behind the scenes. As keys and values are added to an empty root node of the tree, when a key conflict happens, another node on the second level is created and the key is added to it instead. The tree will get bigger and bigger as items are added. Linking a second level node by reference Imagine that we want to mutate this tree. We run the algorithm to figure out where to put the key, then need to copy the root node (it’s immutable, so we can’t mutate it). This is where the magic happens. We don’t have to copy the second level of the tree. We can re-use it. We just need the reference. In a mapping with billions of keys, perhaps five nodes out of 10,000 change. This is called structured sharing. The algorithm is called HAMT (Hash Array Mapped Trie). Yury implemented it in the Python Standard Library. It can be found in contextvars module. The hamt.c file has a 200 line comment explaining in detail how this data structure works. How we can benefit from it All subinterpreters run in the same OS process so they share the same memory space
If we want to access this tree from the main subinterpreter, we just look into the memory directly. It is immutable and will not change.
If we want to add a key, we don’t have to copy the entire tree, we just create the missing new branches and reference others
If we have a tree with billions of keys, we just create a couple of tree nodes in a worker subinterpreter, the rest can be re-used.
Key: using immutable things, we can access the underlying memory safely without locks, as long as we can guarantee that the data would not be garbage collected.
memcopy is used and is really fast. Create tree nodes in the worker subintrepreter and re-use the rest Yury performed a benchmark. Structured sharing is significantly faster than using pickle, potentially 6x to 150,000x faster. Yury talked about some of the implementation details icluding incref and decref. incref keeps Python objects alive when the ref count goes to zero. The data sharing mechanism has to guarantee safety. The hard part of this is implemented in memhive, but some bugs need to be fixed. Do not use it in production. Yury is building this for Edge DB to optimize cloud deployments. ???? back to top
Measuring the performance of CPython¶ “Measuring the Performance of CPython” by Michael Droettboom (Microsoft CPython Performance Engineering Team, Faster CPython Team) I did not watch this full talk, because I left early to go to the PSF Members Luncheon. A few key takeaways from what I did watch: The Faster CPython Team has made Python 3.11 20-60% faster depending on what you are doing
This talk was born out of a need to benchmark in order to determine if they were working on the right things to impact Python performance
In the Python world, benchmarks are in pyperformance suite (a little over 100 benchmarks, some 30 or 40 years old)
Not all benchmarks are created equal Other projects that have aimed to make faster Pythons over the years Computer Architecture: A Quantitative Approach has a useful chapter for looking at benchmarks and categorizing what they are useful for Toy benchmarks: simple, less than 100 lines of code, cool, but not a common use case
Real applications: take some code that runs in production, maybe on a massive scale and running it in a benchmark suite
Microbenchmarks (not in the textbook): measuring a narrow feature of the language Sub-topics of real applications benchmarks: Modified application: change it slightly to make it better as a benchmark
Application kernel: taking one subsystem of application and benchmarking that specifically ???? back to top
PSF Members Luncheon¶ I attended the PSF Members Luncheon. I sat at a table that included Naomi Cedar (Author of The Quick Python Book), Tres Seaver (Zope), Chris Brousseau (PyBay), and Phebe Polk (PyBay Piggies). I really enjoyed the presentation. At the luncheon, I had the chance to chat with Jannis Leidel (Anaconda) again. At PyCon US 2019, I met him serendipitously while I was working at the PyLadies booth. Once upon a time, he worked on Pinax, an open source library that I later helped maintained. I was able to learn more about the history of the project from him. I later went over to Hynek Schlawack (prolific open source maintainer) who remembered we had met at PyGotham 2019 where I attended a talk he gave! While chatting with him, he introduced me to Hugo van Kemenade, the CPython 3.14 and 3.15 Release Manager. It was very cool to meet him, and it turns out that he maintains Pillow package. I am a big fan of Pillow. I used it to create my Twitter art bot which is very special to me. :) ???? back to top
Profile Pic¶ My friend Melanie Arbor (O’Reilly Media) takes great profile pics. At DjangoCon US 2017, she took my pic, and I loved it. I’ve used it as my social media profile pic since then, but it’s getting a bit dated. I saw her at PyCon US and asked her if she would be willing to take a new profile pic of me. It turned out that she had already created a “Profile Pic Palooza” open space for Saturday at 3 pm. As a bonus, I got to catch up with Chalmer Lowe (Google) for the first time since PyCon 2019. New profile pic :) ???? back to top
Juggling¶ Not only does Ned Batchelder (coverage.py Maintainer) maintain a powerful Python package and give popular conference talks (check out People: The API User’s Guide), he is also a juggling enthusiast. When I saw his juggling open space in progress, I couldn’t resist stopping. I juggled for the first time in probably decades! It was also great to catch up with Paul Ganssle (Google, CPython Core Dev, pytz maintainer), and meet Rob Ludwick (Bank of NY Mellon) and William Higgins (Consumer Financial Protection Bureau). Video of me juggling! ???? back to top
Hallway Track and Expo Hall¶ It was great to see Jacob Kaplan-Moss (early Django Engineer) and Frank Wiles (RevSys, former DSF President) again. I saw Glyph Kefkowitz’s badge and immediately knew he was probably the same Glyph (Twisted Project Leader) I’d come across on social media. I learned more about Twisted when Amber Brown gave a keynote at DjangoCon US US 2019 “The Natural State of Computers”. Glyph and I talked about the role of Twisted in the async world and the possibility of using Twisted for async orchestration. Speaking of my DjangoCon US 2017 talk, Russell Keith-Magee (Beeware Founder, former DSF President, prolific public speaker) was my speaker mentor at the time. At PyCon US 2024, I had the chance to catch up with him before the PyLadies Auction. I congratulated him on the acceptance of PEP 730: Adding iOS as a supported platform. Python now runs on iOS without a patch! Russell’s Beeware Project enables you to write an app in Python and release it on multiple platforms, including desktop and mobile! Check out his PyCon US 2019 Python Black Swans Keynote) and many Beeware talks. ???? back to top