semi/signal

The Fragment Sync blog is up. Up first is a series of posts about the history of Fragment Sync development, from its original incarnation as Firesync to what it is now. It’ll also give me a chance to present some of the design and technical decisions that went into making Fragment Sync and explain what sets it apart from many of the existing sync solutions out there.

I played around a bit with one of the .Net framework’s compression classes (System.IO.Compression.GZipStream) recently and got some unexpected results. First, the module is a bit weird in how in takes in and outputs data (e.g. for decompression you read directly from the GZipStream, but for compression you read from a MemoryStream), but once you get past this you run into a more significant issue: the compression library is not optimal and, in certain cases, the size of the “compressed” data can be significantly larger than that of the uncompressed input. I did a few ad-hoc and very unscientific tests, but they were good enough to give an indication that there was a problem. GZipStream is probably still good enough when there’s highly compressible data, but for situations where you don’t have any idea of what’s being compressed it’s probably best to look for another solution.

There’s a thread on MSDN about this issue (System.IO.Compression not as good as compressed folder) and also a feedback entry on MS Connect.

These were posted a while back on the Fragment Sync site.

I haven’t put a whole lot of thought into this, so there may be something major I’m overlooking. In any case, I was thinking that virtualizing socket ports per user in a multiuser operating system would be a very cool and worthwhile feature. Doing this would allow applications running under different users to bind to the same port. So, two users running an app with a server component could both accept incoming connections and function simultaneously and independently.

Now, the obvious question here is how would this map to an IP address seen on the local network. A lesson from virtualizing operating systems could be taken here: treat every user as a different machine on the network, so every user instance gets it’s own IP (set manually, assigned by DHCP via. a router, or whatever) and you could (I think) fake the MAC address easily.

There’s the obvious performance loss, but w/ current processors that should be fairly minimal, even for heavy traffic.

I was pretty disappointed today to discover the required login procedure for desktop apps using the Facebook API.

Having a user go to a web browser to do the login then come back to the app is a whole lot of grunt work that could easily be eliminated by having login functionality in the API. Why is this not done? I’m not sure. Security may seem the obvious answer, but that’s assuming app developers are completely inept and won’t properly encrypt, store, or transfer credentials. Even if that is the case, it is the trust between the user and the app that is the basis for security here.

There is a workaround available in the facebook-c-sharp library. However, as explain in this thread, actually using the library will result in your app breaking Facebook’s Developer Terms of Service.

On the subway in NYC, you come across… um… eccentric personalities from time to time. On the F train last night, I was sitting a few feet away from a man who was muttering something to himself, he was loud but I couldn’t make out what he was saying. A few minutes later he yelled out across the train car, asking if the next stop was Union Turnpike (it wasn’t). A woman responded that she didn’t know, and he proceeded to explain to her that he was happy, very happy; in 2006 his doctor has diagnosed him with terminal cancer and told him he had 2 months to live. However, his cancer was now gone; he had cured himself by regularly drinking a mixture of green marijuana and ginger.

So there you go, cure for cancer = green marijuana + ginger

Below is an email response to my aunt sometime in July, when Cuil was launched,

Yea, I read about cuil this morning. I played around with it a bit and they’re getting quite a bit of press due to the founders past employment w/ Google and their claim that they’ve indexed more pages than google. However, it’s a pretty poor launch: shoddy search results, mismatch between sites and images, useless results “summaries” and terrible performance…

I never bothered with Cuil again until I read this a few days ago,

Cuil isn’t performing well any way you look at it, and I can only imagine how nervous the startup’s management team and investors must be by now. Based on the numbers and graphs we gather from Google Trends, Alexa, Compete and Quantcast, you could even say search engine traffic is nearing rock bottom.

I decided to give Cuil another try to see if anything had changed since my first analysis. Unfortunately, I still encountered the same frustrations; here’s the first result in a search for “fallout 2”,

I think that’s Evander Holyfield in the associated image and, as far as I remember, he was not in Fallout 2.

Attempting to compete with good-to-better search results than google would have put Cuil in a steep uphill climb for success, but such poor results and mis-associations point to flaws in the technical architecture of the service itself. Not to mention simple usability flaws; results in a multi-column layout is a dumb idea – it’s completely alien from the way people actually read (well, read English, at least) – I’m supposed to read top to bottom, scrolling down… then scroll back up to continue?!

What’s amazing in all of this is that Cuil’s founders are apparently experts at search and managed to raise $33 million in funding! … and what’s even more amazing is that they were ranked as one of the most successful U.S. startups by Business Week. Business Week’s explanation for this apparent oddity is as follows,

Cuil raised $33.25 million in that period, and whatever you think of the company or their product, the amount raised indicates that investors judged them worthy of significant backing, as some TechCrunch commenters noted. Obviously, that’s different from them getting traction or becoming profitable. But they made the list under the criteria we used, which we made clear from the start.

I guess you have to buy into the idea that funding = success.

Great graphic from washingtonpost.com,

So, a little story. I was working on some code that periodically spawns off a child process to do its thing, then read the results in the output stream from the parent process. Now, this periodic spawning was being done asynchronously in the parent process via. a thread. All was well and good, but there was also another thread in the parent process which created a file with a temporary name, downloaded and wrote a bunch of bytes into it, then renamed the file to a proper name. Unfortunately, I began to notice that the rename operation was failing as the parent process couldn’t get access rights to the file that needed to be renamed due to a sharing violation. I checked, double checked, and triple checked that I was closing the file, and everything looked perfect. After a bit of headbanging, I figured I should probably check to verify that another process is not holding the file handle. I then ran a wonderful utility called Handle to see exactly what processes had the file handle and, to my surprise, it was a child process that was spawned. Now, this was weird as hell as the child process did nothing with this file or directory or any file i/o whatsoever. After a bit more headbanging, I made the unpleasant discovery of child process inheritance. This forum thread, discussing a similar issue with a .NET’s TcpListener, actually pointed me to the issue.

Now most of this was in C#. One tricky aspect of this issue is that the file i/o mentioned was done in a bit of native code using fopen/fclose. I never encountered the issue with similar code using a C# FileStream. My assumption is that the file handles from the C# functions were explicitly not inheritable, while those created by fopen were. The underlying Win32 CreateFile API function does provide for this feature, but it’s not exposed via. fopen.

If this parameter is NULL, the handle returned by CreateFile cannot be inherited by any child processes the application may create and the file or device associated with the returned handle gets a default security descriptor.

CreateFile ignores the lpSecurityDescriptor member when opening an existing file or device, but continues to use the bInheritHandle member.

The bInheritHandle member of the structure specifies whether the returned handle can be inherited.

Now it was time for the really tricky part, fixing this. I didn’t want to change the native code (which in retrospect may have been an appropriate course of action and much easier to do), so instead I tried to see if I could prevent the process from inheriting the handle. The Win32 CreateProcess function has a bInheritHandles argument that can be set to false to prevent child processes from inheriting handles. Unfortunately, I was using the C# Process class and it provides no means to set such a flag. I eventually P/Invoked the CreateProcess function (this blog entry helped a great deal), but faced disappointment as I discovered that I can’t redirect standard output from the child process without bInheritHandles being set to true. I eventually altered the child process’ code to write its output to a file (which was actually better behavior for the app) and finally closed the door on this issue.

As hinted in my previous post I’m working on some bluetooth stuff. Specifically, I’m working with the OBEX-based File Transfer Profile. I’ve been utilizing my cell phone for all testing and cell phones are the likely target for this functionality in the product I’m working on (more on that in a later post). Having played around the the technology for a few months and written a library for file I/O on top of Windows’ sockets functionality, I have a fairly positive impression of the technology. As with all wireless tech, it’s great not having to deal with cables (especially vendor-specific ones), but more-so it provides a nice bedrock for device-to-device communication, which is something that’s not quite trivial with Wi-Fi.

With my love of bluetooth, it’s become quite perplexing to see such a dearth of devices that actually support it. I’m not referring to cell phones or headsets for cell phones, but other devices such as digital cameras. It could be argued that communication over bluetooth is slower than a USB cable; this is true, but bluetooth v2.0 has a respectable 2.1Mbit/s (respectable in the sense that most people have about the same throughput with a low-end broadband connection), and it’s certainly cheap enough to have a bluetooth adapter in addition to a USB port on a device for instances where convenience takes precedence over transfer speed.

In searching for any bluetooth devices out there other than cell phone and headsets, I came across Seagate’s DAVE; a battery powered external hard drive, supporting bluetooth, wi-fi, and usb connections. Now this seemed like a really cool idea, a completely wireless, external hard drive, in a beautifully small form-factor. The first mention of DAVE seems to be at San Jose’s Tech Museum in Jan. 2007, coming in 10gb or 20gb flavors. The next mention of DAVE seems to come over a year later, in Jan. 2008 at Digital Life, with DAVE now 60gb in size and the unfortunate mention that Seagate does not plan to sell the device directly, but instead sell it to smartphone manufacturers to rebrand and resell (and quite certainly limit compatibility to only their product lines) to users.

Unfortunately, 12 months later, and there’s no sight of DAVE from Seagate or a rebranded DAVE from any smartphone vendors. No Seagate product page exists on DAVE and this cool little hard drive seems to have disappeared from existence.

As for similar devices, I found an old announcement for the Toshiba Pocket Server which seems to have never seen the light of day. There was also the BluOnyx which was shown in early 2007, then a corporate merger (Agere – LSI), then new signs of life, but alas this seems to be another cool product that won’t make its way to market.