semi/signal

The problem

One annoying problem I began encountering with ScratchGraph a while ago was fake signups. Every so often I would notice a new account created but there would be no interaction on the site beyond account creation. I’d also notice some other errors in the application log, as the spam bot would attempt to fill in and submit every form on the landing page, so I’d get login failure and reset account errors as well. At first, I figured I could just ignore this; metrics would be a bit off, I’d have a bounced welcome email from time to time, and I could just purge the few fake accounts at some point the future. Unfortunately it got to the point where there were so many fake accounts being created that figuring out if an account was real took more effort, there was a lot of garbage in the database and logs and, perhaps most importantly, welcome emails being sent out would be bounced or flagged as spam, dragging down my email reputation and increasing the possibility of emails going to spam.

A solution

There’s a bunch of blog posts from email marketing services detailing this issue, e.g. Mailchip, Folderly. There’s usually a common few solutions mentioned:

• ReCAPTCHA
• Email confirmation for new accounts
• Some sort of throttling
• Honeypot fields

I opted for honeypot fields.

• I wanted to minimize dependencies and additional integrations, so no ReCAPTCHA
• Email confirmation seemed like too heavy of a lift and I disliked the idea of having a user jump from the application to their inbox

• Throttling kinda makes sense (I could see if other forms were submitted around the same time, with the same email address, and flag the account), this is possible but not trivial for a PHP application where you don’t have service-level jobs running in the background

So, I added a honeypot field on the signup form. In practice, I wrapped a text input in a hidden div and I gave the input the name fullname. A user’s full name is also not requested/collected during sign up and I figured the spambot may try to gauge what to do with the field based on its name, so I should give it a realistic name. On the backend, if fullname is non-empty, the request still succeeds (HTTP 200), but an account is not created. Instead, email and IP address are logged in a table for spam signups (I figure I might be able to do something with this data at some point).

Efficacy

I was somewhat skeptical as to how well a honeypot field on the signup form would work, I was imaging spambots being incredibly sophisticated, maybe noticing the field was hidden or there was no change on the frontend. Turns out this is not the case or at least not the case for the spambots I was facing. From what I can tell, this relatively simple approach has detected and caught all fake signups from spambots.



I imagine there’s a shelf life to this solution (happy to be wrong) but, when it starts to fail, I can always integrate another approach such as throttling or ReCAPTCHA.

I started learning about microservices and seeing momentum the architectural pattern around 2015. A number of developers I worked with at Grovo were enthusiastic about the idea as a solution to a number of scaling issues we were facing. The more I looked around, the more I saw developers across the industry purporting microservices as the new, modern architectural pattern. For some companies, this was natural evolution in their systems architecture, but for many companies, especially smaller startups, the push towards microservices seemed driven by the idea that, despite the cost and difficulties, this was going to be the status quo for leading tech companies and that was the direction they should be headed in. I was on the “smaller startup” side of the industry and, from my vantage point, this perspective didn’t seem to be based on any sort of real engineering analysis but more-so aligning with what was being talked about in the industry, adopting what larger companies (e.g. Netflix) were doing, and a belief that microservices were a silver bullet to scaling issues. Looking back, none of this is too surprising. Following in the footsteps of Big Tech, even when your business is operating at a different scale and your tech problems look very different, or jumping on the hype train for the latest technology (e.g. the “just-got-back-from-a-conference” effect), continues to be prevalent in the industry.

For VC-backed startups, it’s also worth looking at the state of the market around 2014-2015. Venture capital funding in tech shot up along with the size of the deals. Closing a VC round and being pushed to grow was also not out of the ordinary. I’ll hand-wave a bit here, but I think this also led to larger engineering teams, a corresponding overconfidence in what could be tackled, and an underestimation of the difficulties of working within and maintaining a topologically complex architecture.

The lesson in all this is to look at and understand industry dynamics relative to your company and your role. Software isn’t made in a vacuum and the zeitgeist of the industry is a key factor in how engineering decisions are made and what solutions manifest.

Finally, this may all sound cynical and, in terms of the pattern itself, it sort of is. Working on microservices led me to view them as an optimization and not a general pattern for application development. However, building with microservices was also a powerful forcing function to really look at and tackle inefficiencies in infrastructure, deployments, and code structure. In addition, the company-level initiative to push towards microservices highlighted the dynamics by which decisions are made and driven, from industry buzz to company leaders to individual engineers. What I learned from all of this improved my technical work in areas beyond microservices and that’s what I hope to really highlight in this and future posts.

Goodbye Adobe Air

Despite positive first impressions with Adobe Air, I began avoiding it a while ago for a few reason:

• Air’s focus was very much geared towards Flash development, not HTML/CSS/JS, and I had no interest in Flash development
• A lot of interesting web technologies never manifested within Air (SVG, WebGL) and it looked doubtful that Adobe cared to add anything that might challenge Flash
• Linux support was dropped – a platform dropping OS support is not a good sign
• Native interaction support, a feature in Air 2 which I was excited about, didn’t impress me in its implementation and it was very much geared towards Flash/ActionScript development
• Adobe’s push for Air became more a platform for mobile development rather than desktop development, to the extent that desktop development was pushed far into the background

With all of the negatives above, coupled with the propriety, vendor lock-in, nature of Air, I really didn’t feel like using it for development of anything.

XULRunner / XPCOM

I was still optimistic and interested in web technologies (HTML/CSS/JS) for layout and styling in desktop applications. As I stated previous:

Looking into cross-platform GUI frameworks, I’ve played around with WinForms (cross platform with Mono), Qt, Gtk, and wxWidgets. I’ve been disappointed to various degrees with all of them. It hit me that the most flexible and powerful cross-platform layout and styling framework out there is the HTML/CSS combo. It’s not perfect (e.g. floats, vertical centering) but it’s pretty damn good.

I stated looking at other solutions. I didn’t want to relive my experiences dealing with compiling webkit (though I was, and still am, tempted to play around with chromiumembedded), so I went with XULRunner, which allows for bootstrapping XUL-based applications (e.g. Firefox). XUL is not HTML, and in actuality provides markup to design a UI with native controls, but one control provided is the iframe control which renders and interprets HTML, CSS, and Javascript.

Having XUL for the application interface, you could write a desktop application with the application logic in Javascript, but you’re bound by the same limitations as a web application. XPCOM is a solution to these limitations and allows for interactions with the host system for things such as reading files, running another process, etc. That said, I wasn’t excited to learn XPCOM – it seemed convoluted and I didn’t want to waste time doing a deep-dive into yet another framework and being tied down to its limitations. I figured if I could write the application logic in another executable and have it communicate with the UI via a lightweight (very lightweight) XPCOM-based component, that would be ideal. In terms of simplicity and availability, sockets seemed like the go-to solution for Inter-process communication between XPCOM and anything else.

The Socket Bridge

So I came around to the idea of a Socket Bridge (I had originally played around with it in an Adobe Air project and was able to apply it here as well). Within the XUL application, Javascript-based XPCOM code would implement the Socket Bridge client and the executable handling the application logic would implement the Socket Bridge server, and the 2 could communicate easily.

An HTTP server could take the place of the SocketBridge server, but I felt that was overkill, less flexible, and added an additional layer of complexity as the server then needed to be connected to the application code.

Trilite

As a proof-of-concept, I began working on Trilite, a simple HTTP profiling tool, with the application logic done in C#, that would send a number of HTTP requests to a server, capture the time it took to get a response, and calculate some simple stats about the results. A pretty simple application but something pretty handy for optimization work.

I’m pretty happy with the results thus far, particularly with regards to having a consistent, stable, and cross-platform UI.

You can find the current code in the Trilite repository.

There is no bootstrap to launch the Socket Bridge server and XULRunner app, they need to be executed manually for the application to launch:

• Launch the server by running /trilite.public/app-server/trilite/trilite/bin/Debug/trilite.exe
• Launch XULRunner, in /trilite.public/xulrunner, with the application.ini file in the root directory. For Windows, you can also run the trilite shortcut in the root directory.

A few screenshots under Windows:

And here’s Trilite running under Ubuntu:

This post should, hopefully, provide a top-level overview of the application architecture. I’ll be writing more about XUL, XPCOM, the Socket Bridge, and Trilite in subsequent posts, providing more details and code.

Something I’ve been doing for a while in my web development work is applying separate classes, interaction classes, to DOM elements that interact with Javascript. Basically, an interaction class is applied to any DOM element touched by Javascript code – an element bound to an event handler, an input element with it’s value being read or written, an element selected for animation, etc. The goal being to de-couple styling from interaction, allowing style changes to not interfere with JS code, and vice-versa.

Below is a bit of code to demonstrate what I’m talking about. As a convention, I apply a “ia-” prefix to my interaction classes.

<a href="#" class="btn-primary ia-begin-testing">Begin Testing</a>


• btn-primary has the CSS rules for styling the anchor as a button
• ia-begin-testing is bound to a JS event that triggers some arbitrary “begin testing” action

If the future, if I want to change the button to a link (remove the btn-primary class), change it to a secondary button (btn-primary to btn-secondary), or change styling in any other way, the Javscript code is unaffected and requires no changes.

In addition, the ia-begin-testing class can also be applied to other elements (another button, a link, an anchor wrapping an image, etc.) and is automatically bound to the same interaction functions, without writing additional JS DOM selection code. The ia-begin-testing class can also be removed, or changed to another interaction class, and the styling on the button remains the same.

While IDs and data attributes are also good choices for architecting this sort of style/interaction separation, I like classes for 2 reasons:

• Selection via class is relatively fast across all browsers compared to selection via data attributes
• Compared to IDs, classes can be re-used allowing the same interactions to be shared by multiple elements (e.g. a button and a link can both trigger the same function)

One of the reasons I wrote this post is as an alternative to the the “grouping of selectors” approach presented in Chris Coyier’s Can You “Over Organize” JavaScript? article. With interaction classes, there’s little need for grouping selectors. Aside from the benefit of de-coupling styling and interaction, you get the advantage of a single class (ia-whatever), on whatever and however many elements, mapping all said elements to their necessary JS functions. With grouping of selectors, some sanity is brought to the scattered DOM selection code, but you’re left with the burden of maintaining a pool of different element IDs, classes, etc.; a chore that only gets harder as the codebase grows and changes.

It’s interesting that in 3 distinct domains I’ve run across the same underlying basis for optimization:

• Graphics: Modern GPUs depend heavily on batching primitives, typically triangles. Instead of rendering triangles individually, you get much better performance by batching primitives together in a list, sending it to the GPU via a single call, then letting the GPU pipelines to do their thing. Even before modern GPUs existed, graphics cards supported techniques like BitBlt which, essentially, performed operations on batched blocks of pixels, to take advantage of the embarrassingly parallel nature of computer graphics.
• Relational Databases: Issuing lots of small queries can kill performance. A better strategy is, usually, to issue fewer queries, joining and returning as much data as possible with each query. Even if these queries becomes complex and costly, the cost of a complex query will usually still be less than the aggregate cost of numerous simpler queries.
• Networking: The speed of light sucks… server and packet switching latencies make things worse. I usually assume ~50ms baseline latency to send a request packet + get a reply packet back from an internet server (I use the term “packet” loosely, referring to programmer-defined, application-level “packets” or messages, or whatever you like to call them, not necessarily TCP/IP packets). Note that this baseline is regardless of the amount of information in a packet and is bound by the travel time between server and client. So, to optimize communication and bandwidth, a good strategy is to transfer as much as possible per-packet instead of depending upon numerous requests/responses to/from a server, which would mean lots of packets and lots of wasted time.

Fragment Sync has an automated update system called autobot. The system is used both for development and public releases (I should mentioned that automated updates to development copies of a networking app is a beautifully amazing thing). Updates are published to the autobot server using a simple publisher app, which puts the update package on the server and calls a server-side script to enter the patch details into a database. One annoyance with the publisher is that I would have to enter details (version, whether or not update is public, etc.) manually. This isn’t too bad, but on more than one occasion I’ve screwed things up by accidentally entering the wrong version or packaging an old executable.

I finally decided that the best way forward was to have the publisher automatically query the executable for information. You can embed this sort of information via. the application manifest, but using the manifest really didn’t cross my mind and I’m not a big fan of manifest files in general. In any case, what I did was have the executable take arguments and output the queried data to the console. This is a bit trickier than it sounds because, for whatever reason, a GUI app can’t send messages to a console without some hackery with the AttachConsole() Win32 function. (I’m dealing with C# and WinForms, but native Win32 GUI apps don’t have a default console either, so this isn’t a WinForms specific thing). The publisher redirected the standard output stream and read in the data that was pushed out.

This seems to have worked out nicely and has pushed me to think about other information an app can output about itself, lending to better communication between related apps.

Tricky software engineering problem. I’ll abstract it and explain the situation.

(fyi, a lot of this is just me thinking out loud, I don’t distinguish between type and instance as I think it’s clear from context and I use the term “flavor” to refer to objects which share a common base)

We have a certain object (ObjectA) that may contain an instance of another object (ObjectB). However, the constructor for ObjectB needs information about its “holder” (ObjectA), so the constructor for ObjectB must accept a reference to ObjectA.

Simple enough to use: create ObjectA; create ObjectB passing in ObjectA reference to constructor; call method to assign ObjectB to member of ObjectA.

However, there are many flavors of ObjectB (a plug-in architecture, with ObjectB as an abstract base class), so we have ObjectB1, ObjectB2, ObjectB3, etc.

Still no problem, we just create the flavor of ObjectB. As they are all derived from the same base class, we can pass them around generically (i.e. no problems with assignment to member of ObjectA).

We also want a single function (FunctionY) for creating our instances of ObjectA, that will automatically create and assign any flavor of ObjectB to ObjectA.

Now we run into a problem. Realize that we can’t pass a flavor of ObjectB as an argument of FunctionY (creating an instance requires ObjectA, which doesn’t exist as yet). So, how do we pass a flavor of ObjectB as an argument of FunctionY?

A simple “solution” is to change the problem and instead of passing ObjectA into the constructors of ObjectB flavors, have some sort of method (Init or whatever) that takes an ObjectA reference and can deal with the necessary ObjectA information after our ObjectB flavor has been constructed (i.e. pass ObjectA to ObjectB flavor after ObjectB flavor has been constructed). This works, but the problem it presents is that the ObjectA information may be important (perhaps even critical to construction), and having a lax policy such as this can lead to cases where ObjectB never receives its instance of ObjectA. So, the code becomes more error-prone. However, despite being more error-prone, this may work in a lot of cases as long as you can wrangle away the ObjectA-specific code from the constructor.

Of course, if we resolve to change the problem, we can also get away with having multiple functions. Or mangle things in any which way we want to get the square peg to fit the circular hole. However, I assumed I had a legitimate problem and decided to look for a solution.

Another simple but, this time, real solution is to create an ObjectB factory (woot! design pattern!) and pass some sort of representation (e.g. enum) for the flavor of ObjectB we want. This representation is passed to FunctionY, which will query the factory and get an instance of the flavor of ObjectB we want. This will work, but I don’t think it’s an elegant solution. We’re creating another construct to facilitate something which should be trivial (in my opinion, at least). We’re also increasing our maintenance burden; every time we create another flavor of ObjectB we need to update the factory and update the list of flavor representations.

I thought about this for a while and realized the central issue here was how to pass type information between functions.

So, I tried a solution involving reflection (this was in C#). Reflection works, as you can dynamically pass the type and dynamically invoke a constructor for the type. However, I don’t particularly like the reflection solution. There’s, of course, the high performance cost, but also the solution was too generic as your making assumptions about what parameters the constructor of a type will take and, as as it’s all dynamic, there’s no compile-time type checking. However, that being said, the bottom line is that a reflection solution works.

Hunting for another solution, I decided to experiment with generics. The idea was to create a builder class that took the flavor of ObjectB we wanted as its generic-type parameter. There would be a public method, BuildIt(…), which would take an instance of ObjectA and would simply instantiate the flavor of ObjectB we wanted using the passed-in parameter as its constructor argument. This would work, expect for the fact that generics don’t allow you to instantiate an object of the generic-type with constructor arguments. I quickly discovered the…

Cannot create an instance of the variable type ‘T’ because it does not have the new() constraint

error message spit out by the compiler. Damn.

If this was C++, templates should free us from this problem, but porting all the code over to another language isn’t exactly a feasible solution, to say the least.

In the end, I’m just kind of bummed out that I could find a solution I was really happy with. I’m sticking with the reflection solution, as this isn’t performance-critical code and I simply have to move on to other stuff.

Finally, looking at things from a wider perspective, just two things that seem weird. First, why can’t types be passed around and used, but resolved statically (by a compiler or a pre-processor). Like how #define is used in C I guess. Also, languages with generics/templates peg themselves as object-oriented, but when dealing with types in a generic fashion they’re completely agnostic to any form of inheritance and, as such, the object’s class hierarchy.