base64 Encoding Pitfall

A picture is worth 1000 words, but let me start with the context. We are updating some config files in our deployment and we need base64 encode our user:pass as the token. My team isn’t allowed to do our own updates to production; we’re required to schedule releases with DevOps and provide them with a RunBook of step-by-step instructions. Here’s what I wrote, and note that I’m writing this piece because there was a process failure:

Image of python instructions to base64 encode a string.

My intern was able to follow these instructions to curl the API endpoint, so I figured all would be fine when seasoned professionals put their hands on the instructions. I suppose “figured” in the last sentence is a synonym for “assumed”….

DevOps notified us that they completed the process. I identify that we have a failure connecting to the service that the token is supposed to grant access to. When we asked about the JSON that was added to the config file, we’re told DevOps copied the return value from the encoding:

Image: screenshot of the response from DevOps that they did not copy the return value as directed.

By which they meant they copied all of “b’SooperSekritToekin'” from above.

As the author of the doc, where could I have been clearer? Would color have helped? I probably should have added a step (despite always getting pushback for doing so) that would have eliminated the need for the user to selectively copy something. For future reference, this is what the base64 encoding instructions will look like:

$ python3
>>> import base64
>>> t = base64.b64encode(bytes('username:password', 'ascii'))  n.b. do not forget the colon between
>>> print(t.decode()) 
# SooperSekritToekin  <-- copy this
base64 Encoding Pitfall

Fixing Pinterest’s Pin Builder

I’m trying to get my kitchen and dining room remodeled. The designer wants me to save ideas to Pinterest. Sure, no proble…. wait, what’s this?

I’m trying to save an image to my board as a new pin. That worked last week. Tonight, all it will let me do is add an image from my computer. But the ‘Save From Site’ button is still there. Clearly something is amiss…

So I fixed it. Locally anyway. It looks like someone moved the input field outside of the parent div and that made the input think it had ALL the vertical room, instead of sharing with the button.

I made a board with screenshots, here:

If you know how to contact customer support at Pinterest, or have an email to a dev on this team, let me know. It probably should be triaged as a blocking bug as it’s a regression and there’s not a workaround… but it’s an easy hotfix.

Fixing Pinterest’s Pin Builder

Pro-tip for Interior Designers

My partner and I are looking to remodel the fairly standard mid-century house we’re in. I’d call it a cratfsman, minus the more charming aspects of the textbook examples. Actually, I call it “Seattle two-story blueprint #4.” It was the brainchild of a guy who was sufficiently knowledgeable, but as we’re learning he didn’t have great follow through. There are a lot of places where he obviously stopped following any kind of plan and left something as “good enough.” What’s prompting this remodel quest is that he really, really liked to keep his spaces separate. Even the dining room and living room, which are connected by a 10 foot opening, feel like they’re entirely different spaces.

This means we’re looking for a designer. Which is hard. We paid several hundred dollars for a walk-through consultation that resulted in: move the kitchen to the other side of the house, take out all the interior walls. When I said “we don’t have a half million dollars to remo the upstairs” they replied, “you don’t have half a million dollars worth of square footage. This is $300K, tops… wait, are you saying there’s a downstairs?” Yikes. Now that we’re back to this project of renovating our forever-home we’re all over the internet.

So I clickbaited that this would be a tip, it’s really a plea: designers, please don’t exclusively feature the chateau, the log home, the mid-century modern with cathedral ceilings… I guarantee 90% of your potential clientele are not in that position. I get that you only have to design ONE house each year if it’s on Mercer Island, but goddamnit I want my house redesigned, too.

I want to see what you did in a kitchen with just under eight feet of vertical space. How did you turn a 200 sq ft kitchen from one-ass to the room where friends and family can congregate -while- dinner is being prepared – without tripping over the guests? What did you do for the living room that always feels small, despite being 300 sq ft? How did you manage the odd angles of a half-assed “breakfast nook”?

So if you could do me this favor, I’d really appreciate it. And I’ll hire your cousin the contractor.

Pro-tip for Interior Designers

Comparative costs for 3D prints -or- why high school chemistry is useful

Greetings! I recently purchased a resin 3D printer and got to thinking, “how much -does- it cost to print a miniature?”. I printed the same* mini in both resin and PLA plastic to get this started. It’s a skeleton from Fat Dragon Games that is designed to be printed without supports. I weighed them on the same scale, which displays three decimal places, so I’m fine taking an accuracy of two decimal places. (*Ok, they’re not 100% the same. the resin mini -does- have the thicker spear, and I think the skull was reworked a little as well. This is the upgraded model from FDG in response to people saying the end of the spear was breaking off during PLA printing.)

Resin on the left, PLA plastic on the right

Let’s start with the easy case, a FDM printed mini. FDM printers operate by pushing a plastic filament into a block that has a heater (to melt the plastic) and a nozzle (to direct the flow). The hot end deposits a layer of filament in very much the same way that frosting is piped onto a cake, then moves up a fixed amount to do the same with the next layer. This mini was printed at 0.12mm (120 micron). With FDM, there are no losses to speak of when I print a mini directly on the build plate, and if there are supports required, I can easily weigh those to consider them a part of my final cost.

The PLA-pro that I buy is currently $23 on Amazon (affiliate link if you want to check today’s price) and the rolls are 1kg. Simply moving the decimal place indicates 1g of filament is 2.3cents. Hooray for easy math. The PLA print weighs 1.97g so the PLA mini costs roughly 4.5cents. Four. Point. Five. Cents. This means for the price of three molded plastic minis I can print an army of about 180 skeletons.

Resin prints are formed from a bath of the resin goo. The build plate descends into the reservoir, UV light cures a layer of resin to the build plate, the plate moves up then back down to within some small distance from the bottom of the reservoir, UV light again and the process repeats. For this skeleton I printed each layer at 70micron (0.07mm). When the print is finished there’s some amount of resin that needs to be washed off the mini, and there are some losses when the resin is filtered during its return to the resin bottle. I’m going to say this is an ideal project where “only a little” of the resin spills out of the reservoir during cleanup, plus the film of partially-solid resin that surrounds the mini and is removed during post-processing combines to be 20% of the final weight. (This is partially based on conversations online regarding resin print costs in industry). I don’t know that’s an accurate value, but it’s a) within reason and b) easy to work with for math.

One thing I didn’t consider when I bought the Anycubic Photon is that resin, a liquid, is sold by weight? volume? Umm… I’m not sure. I know I pay $24.99 for a bottle, but the Amazon listing shows it’s both 500ml and 500g. Typically resins are sold by weight, and we found the cost of the PLA print by using weight, so let’s do that again, and we’ll say this resin costs $25/500g or 5cents/g.

For those of you a little rusty with your chemistry, density is mass divided by volume. 1ml (milliliter) of water ideally weighs 1g (gram), which means its density is 1g/ml. Note that 1ml is 1cm^3 (which will be handy to remember in a minute). Intuitively I considered that UV curing resin cannot have the same density as water, simply because there are lots of things that make up this goo. So I looked up the spec sheet to find that it has a liquid density of 1.1g/cm^3 and a solid density of 1.184g/cm^3.

So, let’s figure out how much this skeleton cost in materials. I have a mini that weighs 3.22g and has a reported solid density of 1.184g/cm^3. But I didn’t put solid resin into the reservoir, which means I need to do a density calculation if I want to be accurate about this. And let’s be honest, we’re talking about 3D printing miniatures for Dungeons and Dragons… we’re freaking nerds so we’re curious about the accurate measurement!

I need the volume of the mini to find the weight of the liquid resin so I can figure the cost per gram. density is mass/volume, so volume is mass divided by density. For the solid: 3.22g / 1.184g/cm^3 = 2.72cm^3. Now I take this volume and multiply it by the solid density to get the mass of liquid resin: 2.72cm^3 * 1.1g/cm^3 = 2.99g liquid resin.

Gut check – does this make sense? I assume the volume stays the same, only the density of the material is changing as a response to exposure to 405nm UV light. A more dense material for the same volume will weigh more.

Let’s add 20% to the liquid mass as described above to get our total liquid mass used in this project: 3.58g. At 5cents per gram, the resin skeleton cost a whopping 17.94cents. That means I can only print 47 skeletons for the cost of the Reaper skeletons listed above.

Rhetorical question: is this a reasonable comparison if we’re only printing 28mm miniatures in the Dungeons and Dragons medium- to large-size? Or, which printer should you buy if you only want one?

The Ender 3 that I listed above has a couple upgrades, in particular to the build surface. I prefer a glass build surface, and I upgraded the extruder gear (hob) on my machine. The machine linked is roughly $260 and I added about another $40 in parts to mine (here’s a link if you want a more bare-bones machine if you also plan to do these small upgrades).

My Anycubic Photon was purchased on sale for $340 and I have added nothing to it. I continue to use the included USB drive despite knowing it’s probably garbage, and I’m still using the resin and FEP sheet that were included in the box. Today, the machine is closer to $450.

I don’t think there’s a reasonable argument that the resin print does not look significantly better than the PLA print at the distance that picture was taken. This print is 100% on par with models that are available commercially, and I dare say they’re better because the spear isn’t all wobbly (I’m looking at you, Reaper). The resin prints cost more, oddly because the prints weigh more, not due exclusively to resin costs (though that’s what I expected coming into this). The resin printer’s layer height has a useful range of 20micron to 100micron. The print time on the resin skeleton at 70micron was about five hours. But! I can fit ten six* skeletons on the print bed and I can print them all at the same time for no additional time cost; the printer illuminates everything in a layer at the same time. On the other end, large prints, like a two-inch-tall section of terrain will still take five hours, at 100micron and I’ll have to add development time to hollow out the model and add holes to let uncured resin seep out.

* I did check how many minis I can print on the Photon’s bed and six is really the max because I print at a 40 degree angle, leaning back.

The resin printer lives in the garage because the smell is really powerful (think tire store, or cheap Chinese tools). It sits inside a disposable aluminum turkey pan in case the resin spills. I have to wear gloves and eye protection when handling anything that may have uncured resin on/in it. Each print has to go into an isopropyl alcohol bath for initial resin removal then a second alcohol bath to ensure the partially cured resin layer has been removed. Then each print goes into a UV curing chamber for at least 10 minutes, as do the paper towels I use to clean up, and the filter funnels I use when returning resin to the bottle. This translates to roughly another $60 in expenses for post-processing.

The FDM print, compared to commercial models looks pretty good. Put some paint on it, put it on a table in a room that’s lit for D&D and I suspect no one will know it’s a 3D print unless they already knew it was a 3D print. The miniature took roughly 90 minutes to complete. Each additional mini adds about 105% time because there’s time lost waiting for the print head to move from one mini to another. The FDM printer’s layer height has a useful range of 120micron to 320 micron (0.12mm to 0.32mm). This means the mini shown above is about as good as it gets with FDM but I can print larger pieces like terrain without it taking all day for a single two-inch-tall tile. I can swap out the nozzle for something larger, which gives me a higher top-end for layer heights and the slicing software knows it can push a wider bead of filament. The opposite is not true, unfortunately. I spent many many hours trying to make a 0.2mm nozzle print layer heights of 0.04mm (still 20% of the nozzle’s diameter) and could not produce reliable results that look better than what you see above. Mostly the issues came down to the filament clogging in the nozzle.
FDM printers also have a larger range of bed sizes. I printed the pieces for this Tiamat model at 150% of the original, and each wing is twelve inches, which just fit my build plate.

The FDM printer lives in my basement and I’ve had two of them running during game nights in their cabinets and no one knew they were working until I mentioned it during a break. I can handle bare filament and printed filament with bare hands. FDM prints are ready to go as soon as the print head stops depositing plastic.

If I had to have only one I would have an Ender 3, despite really appreciating the larger build volume of my CR-10S. But the Anycubic will be fantastic for printing “special” minis like player heroes, bosses and military vehicles that require high detail to fit the game. If I need more minion pieces, like skeletons, they’re going to be printed on an FDM.

Comparative costs for 3D prints -or- why high school chemistry is useful

Galvanize Data Science Prep, pt 2

Hey, so guess who forgot to write an update last week? The course has been great, I highly recommend it. Part 2 was released yesterday (Wednesday, the course concludes Friday). [edit: and an update to Part 1, see the list below] Pt 2 is intended as a primer for students who are enrolling into the data science intensive, and looks like it expands on the topics covered in the prep course. The instructor won’t be available after Friday, but the slack team will still be active, and the other students have been helpful whenever there’s a question posted. Here’s what’s additionally on offer:

  • Git Primer
    • Primer, Concepts, submitting work to the instructor
  • In-terminal Editors
    • Vim
    • Nano
  • Object Oriented Programming
    • Terminology
    • Classes
    • Initialization and ‘self’
    • Methods
    • Magic Methods
  • Pandas
    • Pandas Series
    • Panda DataFrames
    • Merging DataFrames
    • Split, Apply, Combine data
    • Visualize Data with Pandas
    • File I/O
    • intro to Exploratory Data Analysis
  • Interpret functions as integrals and derivatives
    • Mathematical Limits
    • Derivatives and rates of change
    • Integrals of functions
    • Connection between derivatives and integrals
  • Linear Algebra 1
    • Matrix Inversion
    • Systems of Linear Equations
    • Vector Similarity
  • Linear Algebra 2
    • Linear Algebra from a Geometric Perspective
    • Linear Transformations Overview
    • Rotations
    • Changing Dimensions
    • Eigenvectors and Eigenvalues
  • Statistics and Probability
    • Random Variables
    • Distributions
    • Estimation
  • Back in Part 1 SQL was added some time this week
    • Database Structure
    • Populating a database
    • Writing simple queries
    • Writing aggregate queries
    • Joining tables
    • SQL style conventions

So wow, that’s a lot of ground to cover. It looks like a really good expansion on the topics in Part 1, and I’m looking forward to going through that material.

The modules I outlined in my previous post have been generally good. There are some aspects where there isn’t a clear line between A and B. In a lot of ways that mimics my experience in engineering school; a primary difference being that I now have the web to look for explanations of things that aren’t making sense. Back then I could reread the textbook, review my crummy notes or try the math tutoring center (if it was open and if I could get over my anxiety about asking for help). I’m much better about using the resources that are available to me.

Now I have a plethora of examples to look for one that makes sense to my brain about how a particular math operation should work. There are some aspects of using NumPy that simply need practice and repetition, but fortunately I’m comfortable and experienced with googling my code problems.

So far there’s only been one challenge (end of module) problem that I’ve called shenanigans on. It required using a technique that was discussed and we were given two basic practice problems to see how it works. The question in question required recalling this technique several modules later, then applying said technique to a new method that behaves very differently from anything we’d encountered previously. I was able to figure it out, but it felt like the learning was less about how to use the method and more about deep-diving on problem solving. This question was either a success or not, depending on their intended outcomes.

Having the instructor has been really helpful. I’m new-ish to Python and how the syntax parses. Having someone to pair with to review my code and tidy it up was fantastic.

I’m looking forward to finishing this first probability module today. As I’m able to continue with the new material, I’ll make some new posts to compare these “deep dives” with the modules in the first section.

Galvanize Data Science Prep, pt 2

Galvanize Data Science Intensive, Premium Prep Course (Day 3)

You’re not behind, I am. I’m writing this on day three but it’s the first blog post in what will be a short series.

Last Monday the venue for PuPPy’s Interview Prep Night was Seattle’s Galvanize location. I’ve been there before for meetups and I was so very impressed and grateful for their hospitality toward IPN. Snacks, water, swag (swag!) and discount codes for their Data Science Immersive prep course. Awesome!

My motivations are mostly altruistic. Over the course of TAing classes for University of Washington, at meetups, and as an organizer for IPN I’ve been asked numerous times, “What do I need to do to get into data science/machine learning/AI?” Previously my answer was along the lines of “git gud at Python…. and how’s your math?” I have a slightly better idea of what’s involved in the foundational education now that I’ve seen how this course is laid out.

The GDSIPP is “self-directed” insofar as we’re left to work on the sections at our own pace but there’s an instructor available from 9:30 to 4:30 Pacific Time for the two week course. Each day begins with a roughly 30 minute stand up and a short lesson about upcoming topics. Midday (1:30) there’s another check in meeting that is for q-and-a and the instructor has additional lessons planned to fill that half hour. The video check-ins have been good for me as I was concerned that I would start to wander off if left entirely to self-directed study and communications via Slack.

On day one we’re given access to Galvanize’s “Learn” platform that has eight lessons, each of which are comprised of sub-sections with code challenges, and a final code challenge that tests student knowledge of the topics covered. The lessons are:

  • Intro to Python 3.6
    • Basic data types, flow control (if…else), while loops
  • Strings and Lists
    • String indexing and iterations, string formatting (notably Python 3.6’s f-string wasn’t introduced), lists and list comprehensions
  • Python tuples, dictionaries and sets
    • Mutability, tuples, dictionaries, sets, dictionary comprehensions
  • Python functions
    • Functions with parameters and arguments, function variables and scope
  • Linear Algebra
    • Vectors, matrices, matrix multiplication, rank and linear independence
  • NumPy
    • Create and modify arrays, use boolean indexing, linear algebra operations
  • Foundations of Probability
    • Language of probability, set arithmetic, set notation, dependence/independence, chain rule, law of total probability, Bayes’ Rule, Bayes’ Fallacy, Bayes’ probability trees, combinatorics, multiplication rule

Students are encouraged to reach out to the Slack workspace if they are stuck on a problem for more than 10 minutes, and in the course intro they go over posting etiquette and using threads rather than cluttering the main Slack space with problem discussion. Students are encouraged to help each other without giving away answers or posting working code. Our instructor has been very responsive and I haven’t felt as though students are solely reliant on each other.

I’m starting day 3 with tuples and sets. My goal is to get into the linear algebra section Friday morning so I’ll have a week to pick the instructor’s brain about math concepts that I had difficulty with in college. So far I’ve tackled series and sequence questions without much difficulty, and I certainly understood what the question was asking even if I couldn’t immediately write code to solve the problem.

I plan to touch base at least three more times before the course is finished. I hope you find this series useful, and feel free to ask question in the comments; I’ll do my best to answer in a timely manner.

Galvanize Data Science Intensive, Premium Prep Course (Day 3)

Dutch National Flag Problem and Code Golf

Last night at the Interview Preparation Night meetup that I host, I was asked to implement the Dutch National Flag problem on the whiteboard. The way it was presented to me was: given an array and a pointer element whose value acts as a pivot, return an array that has all the values less than the pointer to the left of the pointer, the pointer, and all values greater than the pointer to the right of the pointer.

I came up with a solution that works, and is fast, but it led to a discussion of expectations when the asker changed the game “oh, it’s supposed to be done in quicksort.”

Here’s all of the relevant code in one spot: If you’d rather not take the time to check it out, I’m adding my comments and thoughts below.

The best implementation depends on the bottleneck in the system.
If I have to do this on a RaspberryPi I’d probably try my implementation and see if there’s still storage space; if not, let’s implement best_dnf. Same if I’m doing this on a cloud computing platform because there’s probably a storage limitation. If I’m in a browser I’d use my implementation because I’ll use all of your system’s resources all day.

These “code golf” answers drive me nuts. Essentially you can’t get to EPIP’s answer unless you a) know it or b) have most of the standard library memorized. I hadn’t seen a,b=b,a before, so I had to look it up:

Python separates the right-hand side expression from the left-hand side assignment. First the right-hand side is evaluated, and the result is stored on the stack, and then the left-hand side names are assigned using opcodes that take values from the stack again.

Dutch National Flag Problem and Code Golf