Yet another week of doing lots … but not necessarily the things I was tasked with. Plus it was my turn to be fire-fighter, which always means a lot of ad-hoc distractions.

While I did some coding, a great deal of time was taken up investigating data issues form our customers. The amazing ability of Dremel to query enormous datasets in realtime is a real bonus. But understanding the meaning behind the results can take a lot of time. One customer was complaining about seeing a huge jump in the number of changes to their project at a certain point earlier this year. I was able to track that down to a misconfiguration that they corrected at that time. For a moment, I was worried that I’d lost data for them!

Another customer was investigating how far back in time our dataset went. Looking at some strange results there revealed a bug in how our pipeline considered some jobs to be “in production”. This was simple to fix, but it required a sequence of three chained changes, each depending on deployment of the predecessor.

Probably the most important part of the week was a meeting with our new director to go over the metrics work we do. Lots of probing questions were asked (which is good!) that made us realise we needed to better understand who’s using the metrics we produce, and why they’re helpful. This is something that it’s easy to lose sight of when you’ve been building for a very long time.

Stats: 30 changes submitted, 53 reviewed



It was quite a busy week, but with not much focus on any one thing.

On the technical front, I deleted two (seemingly) unused fields from a very common configuration protobuf schema. Naturally, I didn’t catch all the field accesses, so there was a round of rollback / rollforwards.

I spent a large amount of time with the rest of my team reviewing old bugs in our backlog. This is something we do periodically, and I find it useful (if tiring). We get review all bugs over six months old in the backlog and see if they’re still applicable. Many times the issue has been superceded and is now obsolete. Sometimes it’s even been fixed!

Stats: 19 changes submitted, 39 reviewed.



Much of my time this week was taken up with a migration. We have a library internally at Google which does exactly what the new UnixMicro() and UnixMilli() in the time package does (but was available many years prior to 1.17). Initially, I started writing a “tip of the week” article about how they can be used. This quickly morphed into a “migrate everyone to the new functions” project. Thankfully, having everyone in a large monorepo helps enormously. I’ve now submitted 96% of the migration changes a couple of days after starting (using the large-scale-change aka LSC process).

Making the initial change was surprisingly simple, using a combination of codesearch and gofmt -r. I was lucky that I didn’t need anything more involved; however, I see that Eli Bendersky has an excellent introduction, Rewriting Go source code with AST tooling which would cover more advanced needs.

codesearch -l 'unixtime\.(To|From)(Micros|Millis)'
while read filename
  gofmt -r 'unixtime.ToMicros(a) -> a.UnixMicro()' "$filename" |
  gofmt -r 'unixtime.FromMicros(a) -> time.UnixMicro(a)' |
  gofmt -r 'unixtime.ToMilli(a) -> a.UnixMilli()' |
  gofmt -r 'unixtime.FromMillis(a) -> time.UnixMilli(a)' |
  goimports > "$filename.tmp" &&
  mv "$filename.tmp" "$filename"

I ended up with a change touching around 4100 files. The LSC tools split this up into around 250 changes, and I worked with an approver to get these reviewed and submitted over a couple of days. I can’t thank my approver enough for the diligent review of such tedious minor changes!

There were a few problems…

  • The main one that cropped up was the result of the goimports call above. In some cases, there were multiple files in the package, but the goimports above was only running on a single file, so I saw some unexpected imports for names that were pulled in from a different file in the same package. These were quickly identified and fixed, as they caused test failures!
  • One problem happened twice: a local variable named time shadowed the newly added import of the time package. This was trivially fixable with a local rename. Twice out of 4100 files is OK to do by hand.
  • In a few cases I discovered areas of code that had remained on Go 1.16 (temporarily). These broke very quickly when the tests were executed. I simply reverted these changes and will revisit them in a month or two.

Overall this is a pretty positive experience. I was able to upgrade a large swathe of Google’s code with fairly little fuss. Nobody asked me to do it — and I didn’t need to ask much in the way of permission. The end result is that code inside Google is more similar to code outside.

The main downside is that I should have been working on migrating a metrics pipeline, and instead I got a bit distracted by an interesting problem. Ah well.

Stats: 267 submitted (244 were the LSC described above), 30 reviewed