Author: shacker

Posted on

Saving Energy At Scale

A while back I promised to write a post describing what I do by day at Energy Solutions, so here goes (with a small twist at the end). But it requires a bit of setup.

State governments often make pledges like “We’re going to reduce our carbon footprint 15% by the year 2020.” To make that happen, they put up a big pile of money and entrust it to power companies, since they understand the energy world. The utilities come up with with incentive programs to do things like encourage hospitals and schools to replace old HVAC systems with newer, more energy efficient ones, or for building contractors to replace lighting systems with smart ones that know when no one is home (to give a couple of examples). They work with organizations like the California Public Utilities Commission to come up with “Measures” — ways to evaluate what kind of equipment saves how much energy in a given climate zone, etc.

It turns out that actually getting a program like that off the ground is hard. There are literally millions of pieces of equipment out there that use energy in some way, and a dizzying variety of ways to measure energy savings. There are millions of buildings in the United States, under the jurisdiction of dozens of utilities. How do you devise an incentive program that’s fair to all, and that actually works? How do you prove that a person claiming an incentive is actually entitled to it? How do you measure the efficiency of a given heat pump or ballast or EV charger relative to its usage in a school, or apartment complex, or whatever? How do you ensure the system isn’t abused, and that the equipment was actually installed? How do you compare the attributes of the relevant Measures against the properties of all that equipment (complex queries at scale)? How do you make sure the person is claiming for the best possible program? And so on.

So the utilities hire a consultancy to design and run the program, write all of the logic that makes the magic happen, and build a web interface and batch processing system to handle all of the data and logic? My company, Energy Solutions, employs experts in every tiny corner of the energy industry, and is one of only a few organizations in the country that knows how to do this sort of thing.

Historically, the company created custom code for each new program (madness!). Four years ago, I was hired to help design and build a meta-system, configurable in every dimension, to host many domains running many programs, on top of all that ever-expanding data. While my personal history is mostly in making content management systems for journalists and academia, this job is different – the website is just a very thin layer on top of the most complex software I’ve ever worked on.

My expertise is in Python, and my framework speciality is Django. Over time, we started adding developers, and I’m now the Codebase Lead for a team of developers who report to a very large (too large?) group of stakeholders. I’m responsible for the quality, security, and performance of the codebase and the system. I do all of the code review, work with the ops team, and spend a little too much time in meetings talking about how to implement the Next Big Idea. So I spend my days bouncing between code, github, Slack conversations, and meetings.

And it’s stressful. For the past few years, I’ve fooled myself that things would settle down and get easier in a few months. But as soon as we solve one hard problem, three more pop up to take its place. I finish my days aching to get away from the computer and hit the trail. In the evenings, I’m brain-drained for hours. Lately I’ve been feeling the need to juggle fewer balls, to do something more focused. I’m honored to function as Codebase Lead, but honestly in need of a break. Recently, another opportunity came up in the same company, and I’ve just made the decision to transition to a different-but-related project. Not fooling myself it’ll be easy, but it should definitely be more focused and have fewer moving parts.

I’m proud of what we do – Energy Solutions does a lot more than just run the project I described there – we’re just a little corner of the company. All-told, ES saves more energy in the U.S. than the state of Alaska consumes (that stat was from four years ago – I’m sure it’s much more now). And they have their sights set high. With the Biden administration’s climate goals shaping up, there’s going to be way more to do in coming years.

A little while ago, I was running a conversion process on some data, watching it all roll by as it ran, and it struck me (should be obvious, but we get sucked in by architecture challenges and lose sight of the bigger picture): This isn’t just a few seconds worth of data – this is energy that was NOT used because of what we do, what we’ve built together. The stress is worth it – we’re making a difference not by talking about the importance of carbon reduction, but by making it happen.

Posted on

Devil’s Slide Bunker

After an amazing hike with Dave at Mount Montara in Pacifica last weekend, headed off to Maverick’s to watch some big-wave surfing (it was flat, despite 6-foot waves farther up the coast!), then stopped at the bizarre Devil’s Slide Bunker on the way back up Hwy 1.

A teenager had figured out a way to climb inside (not obvious!), then up on top. Caught him in mid-air jumping down, before he ventured back in to help his little brother, who was temporarily stuck.

From a distance, I thought it was an abandoned attempt at a modernist home, but found this description in Atlas Obscura later:

“The bunker on Devil’s Peak was originally built during World War II as a triangulation and observing station and was once simply a piece of a much bigger set of buildings and facilities. When in service, a watcher equipped with a set of binoculars would keep watch out at sea and if they spotted any enemy ships they simply radioed a massive six-inch gun not far away which would sink them before they got close. Unfortunately, with the advent of more modern missile defenses the station became obsolete and the entire site was abandoned in 1949, leaving an empty bunker atop Devil’s Slide.”

Amazing piece of history, and a local wonder.

Posted on

Uploads Are The New Downloads

Got a note from my boss yesterday, asking why my voice on a meeting was speeding up and slowing down and sounding squishy. That shouldn’t be – we pay for and receive gigabit bandwidth (which translates to 500mbps over wifi).

Since forever, ISPs have put all the emphasis on download speeds, and given the general public very slow upload speeds. That was fine, until Covid – most people just want to watch NetFlix and have no need for fast uploads. But now we all have multiple people at home on Zoom at the same time, and upload speeds have become the new bottleneck.

Kids playing near an old boat, China Camp. Photo copyright Scot Hacker, 2020.

When you’re sharing video of yourself, you’re uploading – each person using around 1.5mbps. Three people at once takes you to 4.5mbps. Speedtest said I was getting 3-6mpbs — right on the threshold with no breathing room. But wait – our gigabit service should be giving us 35mbps upload speed!

Contacted the provider and discovered that our cable modem was old enough to have fallen off the end of the “supported” list. So even though we pay for good speed, and we blanket coverage through the house and yard with a mesh network (eero), the bottleneck had become the modem itself. If I’d been leasing the modem from the ISP like most people do, it would have been upgraded by them. But years ago I chose to save on the monthly rental cost by owning my own modem.

So, takeaway is this: If you own your own modem and are having a crappy Zoom experience, check all of your bottlenecks, and re-check your ISP’s modem compatibility list. Also, if your partner shows full-length movies to their students while you’re trying to have critical meetings, consider joining via phone until you get a new modem :)

Posted on

Django: Testing for Missing Migrations

When adding or altering model schemas in Django, developers typically generate and commit accompanying migration files. But, counterintuitively, Django wants to track all model changes in migration files, even if they don’t result in database schema changes. Those can be easy to miss.

Regardless the reason, you never want your repo to be in a state where migrations are detected as needed, but accompanying migration files aren’t committed.

Writing a pytest test that can be triggered on github, Circle, Travis, Jenkins, or whatever you use turns out to be trivial, but I couldn’t find documentation or examples on the interwebs, so am posting this here for posterity:

import pytest

from django.core.management import call_command

@pytest.mark.django_db
def test_for_missing_migrations():
    """ If no migrations are detected as needed, `result`
    will be `None`. In all other cases, the call will fail,
    alerting your team that someone is trying to make a
    change that requires a migration and that migration is
    absent.
    """

    result = call_command("makemigrations", check=True, dry_run=True)
    assert not result

It really is that simple! If no migrations are needed, result will be None. Any other return value means someone should run ./manage.py makemigrations to see what’s missing, and commit the results.

Posted on

El Radio Fantastique

Cycling Pt. Reyes/Petaluma today, stopped off to check out a trailhead I hadn’t seen before, when suddenly I heard a band. Just over the hill, this family – Mom and Dad and two daughters – shooting a music video out in the middle of nowhere. “El Radio Fantastique” they call themselves. As Beefheart said, “Practice in front of a bush.”

Felt a bit embarrassed when I realized too late that they were in the middle of filming a music video – I had thought the guy with the smartphone was another hiker surprised to find a band playing on the trail! My apologies to the band if I interrupted your creative flow.

More about the band here.
Listen on Apple Music here.

Posted on

In Defense of Flickr

Update: This post has been substantially updated and re-posted as the Medium article “Flickr Is No Ghost Town” – please read that version instead of this one!

Original post:

People like to say flip things like “Flickr is dead” just like they’ve been saying “Apple is dead” for decades. It’s true that Flickr’s heyday has passed, but there are still hundreds of thousands (millions?) of photographers posting there daily. Interaction is lively, there’s an interest group for every photography niche you can think of, and users are really supportive with the compliments and CC. Flickr still has upwards of 75 million accounts, and can receive up to 25 million photo uploads on a good day (stats).

Sunset at Berkeley Marina

Flickr dominated the online photo sharing scene for around 15 years. But as Facebook’s popularity rose, FB became the world’s dominant photo sharing platform, eclipsing Flickr’s numbers (which were already astounding). After a recent brush with financial ruin, Flickr was purchased by SmugMug, and they’ve been great stewards of the platform so far, with improvements being released on the regular.

Sunset at Berkeley Marina

Somewhere in there, along came Instagram to soak up much of the remaining photographic juice in the room. Everyone is on it, so you’d be nuts not to use it, right? I use it too.

So why not just use Facebook and Instagram and call it a day? A bunch of reasons!:

  • Instagram images are tiny postage stamp versions of the images you’ve put so much work into. It’s almost an insult to your photos to display them so small with no full-size web option. But we all do it because we have to (and it’s fun).
  • Instagram has hashtags, but no real “groups” – no organized photo communities.
  • Facebook does full-screen, but they still compress and rewrite your images on upload, even if you enable the HD mobile setting.
  • Not everyone is on Facebook. Millions of people won’t use it for either personal or political reasons (I’m sure we’ve all seen numerous friends leave the platform in the past couple of years). People without FB accounts simply can’t see your work here. Out of bounds. Walled gardens have their place, but I don’t want my photos in one.
  • What if you decide to leave Facebook in a couple of years? What happens to all the work you posted here? Or will the fact that your photos are on FB prevent you from leaving the service even if you want to for other reasons? You’re “locked in.” Putting your images on Flickr instead means your images are decoupled from your social network, which gives you freedom.
  • Both Facebook and Instagram strip out all of your EXIF data, while Flickr does not. I really enjoy studying the EXIF data for other people’s images, or reminding myself of settings that were used on my own.
  • Only Flickr provides full and detailed statistics — not just of likes, but for all views, since the beginning of time (just realized my account is coming up on one million total views since I started there in 2005, wow!)

On Flickr, your images are available in super high resolution, with a wide variety of copyright options. There’s a huge number of interest groups, and detailed statistics. The challenge of getting an image into “Explore” (Flickr’s homepage featuring the best images on the service, changing constantly) is ongoing, and so rewarding when it happens. Discovering great new photographers daily is inspirational. Flickr is photo paradise to this day, IMHO.

Protester returning home, El Cerrito

All of which makes me wonder why so many photographers I meet aren’t using it. It seems like the best of all the options (yes there are other options, but they don’t compete with Flickr IMO, except maybe SmugMug (I’m curious about the features and community there too, honestly – promising).

Tower of Power

Are you on Flickr? And if not, why not?

FWIW #1: I usually post the same image to Instagram and Flickr at the same time, but only share the URL of the Flickr version, since it’s high-res. I only post images on Facebook occasionally, and just for the audience of my friends, rather than for the photo community.

FWIW #2: No I’m not a Flickr employee! Just a long-time user who never fell out of love.

FWIW #3: I’m shacker on Flickr and would love to follow you if you’re there too.

COVID-19 shopper at Trader Joe’s
Posted on

Six Reasons To Use Apple Music Rather Than Spotify

This is a radically shortened version of my Medium piece Spotify vs. Apple Music (Why I Can’t Switch)

I see Apple Music as having six major advantages over Spotify:

  • Better audio quality (but not by a landslide). This is not just my subjective impression – Google it for verification.
  • 100,000 track max library size, compared to 10,000 at Spotify (that’s a deal breaker for me – I could not even fit my “favorite tracks” into a Spotify library).
  • Full-screen album covers instead of being forced to view tiny thumbnails (sorry, but cover art is really important to me, maybe I’m just old school). Though in the switch from iTunes to Music, cover art is now limited to 600px square, unfortunately.
  • Freaking amazing query-based playlists (Smart Playlists). Spotify has nothing comparable.
  • Ability to switch any view between Songs view, Albums view, Playlist view. IOTW you can view a Playlist as a collection of the albums they come from, etc.
  • Vastly superior/easy to use integration of all of your out of print or manually digitized audio content into your cloud space. Spotify makes this difficult if not impossible. This is a deal breaker for people who spent years digitizing their LP and CD collections.

For me, it’s just no contest. I tried Spotify for a month and found it pretty lame in comparison, given that the pricing is pretty much identical. AM gives you a lot more for your money.

Posted on

Beyond vs. Impossible — Fake Meat Taste Test

Used to be that Beyond meats were available in the grocery store, but you had to go to a restaurant to try Impossible Burgers. But recently I found Beyond and Impossible side-by-side in the grocery store, and bought one of each. Last night I set up a little taste comparison test for family.

These modern fake meats are so delicious, it’s hard to find a reason to justify the eco- and health costs of buying real hamburger anymore.

Setup: Two patties the same size. Added the same amount of salt and pepper to each, grilled them at same temperature for the same length of time (3 mins/side). We tasted them as-is: No condiments or extras to get in the way of taste.

Impossible on the left, Beyond on the right.

My wife and son didn’t know which patty was which. To make things fair for my own opinion, I left the room and asked them to flip a coin to decide whether to switch the labels on the plates or not.

Conclusion: We all thought Impossible was slightly sweeter, and sure enough, the label shows that it contains a bit of sugar. We all though Beyond was slightly more chewy, and sure enough, it turns out to contain a bit more fat. Impossible had a bit more native seasoning – a flavor that’s hard to put your finger on, but good. Impossible was also more reddish in color (beet juice, right?) but Beyond had a slightly more realistic taste, if that’s what you’re going for.

At the end of the sampling, we all agreed that if a store had both and they cost roughly the same, we would choose Beyond over Impossible (consensus!). We thought Beyond was more delicious by a shade, despite the fact that Impossible has more seasoning built in. But both are delicious, and it’s gotten really hard to justify the eco- and health-costs of buying hamburger meat when products like these are available.

Posted on

Honest Photography

Almost no one posts images straight out of the camera (SOOC) — there’s always work to do, improvements to be made. But how far can you go before crossing an ethical line?

As our photo editing tools grow artificial intelligence, and tricks that used to be difficult become progressively easier to pull off, photography forums continue to erupt in debates over what kinds of edits cross the line from acceptable and unacceptable? How far can you before an image becomes untrue? I just posted this in the /photography subreddit, on a topic titled “No It’s Not Cheating“:

There is only one question that matters as far as I’m concerned (but several ways to ask it): Is this an honest photograph? And by that I mean “Is the viewer being intentionally deceived?

If you convert to b/w, the viewer is not deceived. If you add a frame the viewer is not deceived. If you focus stack, the viewer is not deceived. If you modify saturation or contrast, the viewer is not deceived.

On the other hand, If you remove a blemish from a human face the viewer IS deceived. If you replace out the sky, the viewer IS deceived. If you change skin tone in such a way as to misrepresent, or “heal” out a passing airplane, the viewer IS deceived.

The difference between acceptable and unacceptable editing is easy when looked at this way. Does this photograph still tell the truth, or does it now tell a lie? Am I attempting to deceive or not? For me, this makes it easy to tell the difference.

I’m shacker on Flickr and Instagram if you want to follow along.

Posted on

Bird in a Bush

The “doldrums” describes the area around the equator where wind sometimes doesn’t blow for weeks, and sailors get stuck, motionless, for long periods of time. Was just thinking about how being on lockdown is kind of like getting stuck in the doldrums. At least we can still get out to exercise, and we have entertainment, but the motion of life is pretty much on hold and out of our hands. Imagine what it was like to be a sailor stuck in the doldrums at any point in the past millennium, just staring out to sea, making up head games to pass the time. We’ve got it good. Maybe it’s time to write a new shanty.

Meanwhile I’ve been doing a lot of photography, and a lot of messing with photo tools and apps, just exploring and experimenting. Yesterday I spotted this little bird in a bush while out walking the dog, and later did this treatment of it in BeCasso.