In search of better tools for NTFS reparse points (junctions, symlinks)

Back in 2006 when solid state drives were fairly new and their sizes were pretty small I found myself with a problem. The drive was too small to hold both the operating system, my programs and all of my documents, photos, videos, etc. While it was acceptable to simply save all of my files to a second, larger, hard drive it wasn’t exactly intuitive the way working with the Documents and Settings or Users folder was.

When I first discovered Junctions I thought I had found the perfect solution. With a junction I could copy the entire C:\Users directory to a totally different drive and then trick Windows into thinking it was still on my C: drive with a junction. After some concerted effort I did just that. The problem with my approach though is that it requires manually recreating all of the junctions and symlinks hidden throughout the C:\Users directory tree. Ouch!

What I have needed for so long was some good tools to make the process easier (and less error prone). Even as of 2014, robocopy still only has bare bones support for symlinks while junctions are entirely broken. So today I am releasing my efforts to dramatically expand the tools available for working with NTFS reparse points on Windows. It’s called ntfslinkutils.

TL;DR

ntfslinkutils is a collection of command line utilities for Windows designed to copy (cplink), move (mvlink), delete (rmlink) and even fix (fixlink) junctions and symbolic links. With these tools you can manipulate every junction and symlink in an entire directory tree and even rewrite the targets of each along the way.

As per usual I am releasing the code as open source under the generous BSD license. You can find pre-built binaries available for download here. Note you may need to install the Visual C++ Redistributable for Visual Studio 2012.

I hope you find these new tools as useful as I do!

Update 2014/9/18: I have removed the Visual C Runtime Library requirement in order for the tools to run under the Windows Pre Installation Environment (PIE). While I have tested this with Windows 8.1 I am unsure if they work with previous versions. Please let me know in the comments if you have a chance to try them on Windows Vista/7.

Working with NTFS reparse points in C++

Probably one of the lesser known features of NTFS are reparse points. A reparse point is a virtual path on the file system that points to some other place. If you’re familiar with *nix systems this sounds an awful lot like a symbolic link which is in essence exactly the same thing. Reparse points come in two flavors; junctions and symbolic links. Both allow you to point to some arbitrary path. So what is the difference?

Junctions were one of the first types of reparse points built in to NTFS. They first saw the light of day in Windows 2000. A junction acts much more like a hard link within the file system with the exception that deleting one will not delete the underlying contents. Their limitation is that they can only point to paths on the same or different local volumes on the machine. You cannot for instance make a junction to a network share nor can you make a junction to a file.

Symbolic links on the other hand remove the limitations of junctions and offer the ability to target virtually any arbitrary path. This effectively makes them junctions 2.0 and on par with the *nix counterpart. However, symlinks can only be created with administrative privileges and some parts of the operating system treat them differently whereas junctions are often treated as if they are still a hard link.

Unfortunately Microsoft have made it pretty difficult to work with both. The Windows SDK provides a rather arcane API in the form of DeviceIoControl to work with junctions and the API has a couple of key missing features when working with symbolic links. In addition, the only other available library I could find was severely outdated and doesn’t handle symlinks (see Windows 2000 Junction Points).

So I decided to write a new library. libntfslinks is a modern C++ library for Windows 7/8 that simplifies working with NTFS junctions and symbolic links. The library’s features are:

  • Create junctions and symbolic links
  • Delete junctions and symbolic links
  • Extract the target path of junctions and symbolic links
  • Determine if a path is a junction or a symbolic link
  • Unicode support
  • 32-bit and 64-bit compatible

The library’s API is written to be as simple as possible without any fluff. To create a junction simply call CreateJunction, to check if a path is a symlink use IsSymlink. That’s it!

The code is open source under the BSD license and available on github along with instructions on how to build and use the library.

https://github.com/caskater4/libntfslinks

Kids Bedroom Remodel – Part 2

In my previous post I explained how I began a project to remodel our second bedroom into a nursery. This post chronicles the installation of 600 fiber optic strands into the ceiling of this room in order to create a night time star field.

Installing fiber optics into a ceiling is no small feat. The project took the help of several friends over long weekends and nights throughout the course of nearly two months. A good part of what took so long were several unique challenges that greatly complicated the installation beyond the typical approaches you’ll find on other blogs.

When I first started to plan the ceiling I thought it would be cool if it could have distinguishable markers much like you’d see in the real night sky. Following some encouragement from my father-in-law I broke out a compass to find the exact direction of North within the room. Then I hopped on my computer and pulled up Google Earth in sky mode.

I set the orientation to match the same angle of the room and started to look for a set of constellations that had a good variety. After I found a good candidate I pulled the star map and began plotting out the other pieces of the ceiling. In order to minimize issues during the installation I added markers for studs, electrical boxes, the crown molding, and access panel for the fiber optic illuminator. Once all of those markers were in place I slightly adjusted the placement of some constellations and stars to ensure nothing would get cut off or blocked. Thankfully not a lot of adjustments had to be made.

Since the fiber optic kit has three different sizes for stars (0.75mm, 1mm and 1.5mm) I plotted 600 points based on the real intensity in the night sky. This means that the largest fibers (1.5mm) make up the major focal points of each constellation. The medium size fibers complete any remaining stars within a constellation and the small fibers fill the remaining space. For the smaller fibers I made sure to use a similar distribution to that of the actual night sky. For example, there is a higher concentration of fibers along the belt of stars that makes up the Milky Way. The resulting map is shown below.

Star Chart

The blue grid matches the rooms dimensions as broken up into 2 foot by 2 foot segments with labels represented as A1 through F6. The long red bars represent the placement and size of the ceiling’s resilient channels (aka: studs). The large red circle in the center is where the ceiling light is to be installed, while the red boxes near the center represent a smoke alarm and fire extinguisher. Finally, the labels W1 through W4 indicate which wall of the room the map corresponds to.

With a complete map of the ceiling finished I began the initial construction. This is when I discovered the first major obstacle in the process. All of the fiber optic kit’s documentation as well as all of the online videos, guides and blog posts relating to it’s installation all had one basic requirement; that you must have attic access to the ceiling. This is so that you can freely drill holes and feed the wiring through the ceiling with a two person team.

Unfortunately for me I live in a condominium. I have no attic access and since I live on the first floor my ceiling is shared with a neighbor above me. The only option in this situation is to take down the existing ceiling and put up all new drywall with the fibers already installed. To further complicate matters my building is constructed with a network of fire sprinklers that I really did not want to start messing with. All of the sudden this was not a project I could do in a couple of weekends.

Thankfully my ceiling was constructed with two layers of 5/8″ drywall. So instead of tearing down the whole thing I decided to take down only the first layer. The reasoning behind this decision was that doing so greatly reduces the difficulty since I would not have to work with multiple layers, insulation and especially that pesky fire sprinkler. Instead I could feed the fiber optics along the ceiling, sandwiching them between the two layers of drywall, until they met at a hole in the upper layer of drywall where the light source would be.

I broke out my Rotozip, set it to a depth just shy of 5/8″, and began to make a series of cuts along the ceiling in a grid like fashion. Once I had a nice square cut out of the drywall I used a drywall knife, a large mallet and good old fashioned strength to pulled the drywall off the ceiling. To my surprise this worked without a hitch.

A word of caution: at first I had made the mistake of cutting the drywall too close to the stud nails. This made it very difficult to pull the drywall down as I was not able to get enough leverage. Soon after I realized that if I made all of the cuts in the center between studs I was able to pry the drywall off the ceiling with ease.

After about a solid day’s worth of effort I was able to get all of the drywall off of the ceiling.

Cutting the drywall in sectionsPulling down drywallDrywall removal completeMy next step was to start drilling holes for each of the fibers in the drywall. I lined up each piece along the wall and began marking them with the same grid and labels I used on the star map. I also marked where the stud lines were with blue tape and added the additional markers for the various holes required.

Laying out drywallMarking wallsOnce done I needed a way to plot each of the star points onto the drywall. To make things easier I used a short throw projector to overlay the star map I made onto the drywall pieces. This is where the grid of blue lines came in handy as I was able to exactly match the 2’x2′ sections of the map to the drywall.

Star Map ProjectionOnce all of the holes were drilled the real work of installing all 600 fibers begun. This was by far the most difficult part of the process and the most time consuming.

The next complication I encountered was that laying each fiber between both sheets of drywall creates a sharp 90 degree angle. This immediately creates a kink in the fiber resulting in less light transmittance and increases the likelihood that the fiber will break altogether. To get around this issue I dug out grooves on the back side of the drywall, creating a 45 degree angle that more easily allowed the wire to safely bend flat onto the backside of the drywall.

Each fiber was then pushed through the drywall, taped down and run in a series of paths along the back. These paths all lead towards the center of the room where a hole in the upper layer of drywall contains the light source. Again, to prevent kinking and breaks in the fiber each one was laid perfectly flat side by side so that when the new layer of drywall was secured tightly against the upper layer no problems would occur.

While running the fiber along the back great care was taken to ensure that no fiber ran over an area that might cause damage from a stud screw or drilling while installing the fixtures.

An alternative method would have been to install a small spacer between the two layers of drywall. However, after much research I discovered that doing so actually increases sound transmittance through the wall (and I don’t particularly want to hear my neighbors any more than I already do).

Installing fibers 1Installing fibers 2Installing fibers 3After every fiber was installed into the drywall and run along the back they were ready to put up on the ceiling. I started with the outer pieces of the ceiling, closest to the walls. This made it possible to tape the remaining fiber across the upper layer of drywall in order to meet at the light source. Where it was feasible I also cut small hopes in the upper layer of the drywall and fed the fiber across the inside of the ceiling using a steel fish tape.

Attaching the drywall

Drywall install completeFinally! A new layer of drywall was finished containing all 600 fibers. It didn’t take long before I was able to hook up the light source and test out all the handy work.

Seeing StarsA huge thanks to everyone who helped with this part of the project; Chris, John, Jen, Matt, and Mike. Stay tuned for my next post covering how the ceiling was finished with texture and paint.

 

Kids Bedroom Remodel – Part 1

Back in January my wife and I decided to remodel our second bedroom as it was soon to be the new home to our then unborn son, Oliver.

My wife’s ideal room had wainscoting, a vertical striped wall paint pattern using gender neutral colors throughout and decorated with classic furniture. Further decor was to be inspired from beach houses. My ideal room was a similar classic treatment wholly inspired by John, Michael and Wendy Darling’s room from the Disney animated classic Peter Pan. Thankfully these two ideas were not so far from each other. It did not take long before we were able to come to an general agreement on the approach.

After more thought I realized that the primary reason the Darling’s room inspired me had less to do with the design of the room itself and everything to do with it’s location in the city, it’s view of the night sky and the magic behind their discovery that just beyond the windows was this magical world called Neverland.

What I really wanted was to somehow capture that magic. I wanted our kids to feel like they could jump up and fly towards the “second star to the right” just like John, Michael and Wendy did. What I needed was a sky for a ceiling, one full of bright twinkling stars.

So that’s exactly what I set out to build. This is the first part in a series of posts that chronicles my adventure in remodeling our kids’ bedroom.

All projects start with good planning and this one was without exception. The first thing I did was start to sketch what the overall room would look like. After a few hours I came up with the following.

Kid's Bedroom Remodel Sketch

The next task was to figure out how to make the starry sky ceiling. The easy approach would have been to buy one of the many night time projectors available but I didn’t like that it would need to be placed in the center of the room to be functional and have dangling wires through the room. I also wanted something more permanent.

The first idea I had was to install the crown molding slightly lower than the ceiling and use the space to hide a series of small projectors that would shine the stars from an angle, perhaps using a network of small mirrors where needed. This approach seemed faulty however. It was very unlikely I could find a projector that could work at such a tight angle and fit into such a small cavity behind the molding. Not to mention the power consumption that it would cause.

My next idea was a network of LEDs embedded across the ceiling. It didn’t take much additional thought to figure out how daunting this approach was going to be. There are simply far too many ways to mess it up to make it worth while, such as incorrect wiring, missing even just one resistor in the circuit and accidentally shorting the whole thing with paint. Maintenance would also be terrible should any part of the circuit fail over time or an LED die.

After lots of searching online I finally discovered a far better approach; fiber optics. To make things even easier I found a company that even make fully assembled kits exactly for what I wanted, covering a ceiling in bright twinkling starry light. Eureka!

With all of the design ideas figured out it was time to get started.

Relocating Windows User Data

Quite some time ago I wrote a post about how to relocate the C:\Users and C:\ProgramData directories of a Windows installation to another drive. This is useful for those of us whom want to use a different hard drive for the operating system and keep documents elsewhere. In my case, my OS is installed to a solid state drive that is simply too small to fit all of my documents, music and pictures. I simply needed a better way to store all the data. When I discovered NTFS junctions (aka: reparse points) I realized I could do just that. I could finally put some breathing room between my operating system and my data.

Now that Windows 8 is here (well almost) I realized I was going to need to move all that data over yet again. In previous years, relocating user data was a pain, specifically because of all the reparse points that have to be fixed recreated once the data is copied to the new drive. Making sure you recreate every reparse point can be quite challenging. Moreso, if you miss one it could mean very bad things when you try to boot the system up again. This got me thinking, why not just write a program to do all that dirty work for me? And so I did.

The program is called MoveUserData. It runs at a command prompt and is intended to be used in the Recovery Console (so that security rights are not an issue). Below you will find the program and full source code.

MoveUserData_v1.zip 
MoveUserData_source.zip

Using the program is quite simple. Copy MoveUserData.exe to a USB drive or CD. Boot the Windows installation disc and go into Windows Recovery mode. Choose Command Prompt from the list of recovery options. Once you are at a command prompt put the CD/USB drive containing MoveUserData.exe into the computer.

Assuming that your Windows installation drive (normally C:) is currently listed as K:, your data drive is D: and that the CD/USB drive is set to E: then you can execute the following commands.

The program will do the rest. Once done, just reboot!

Disclaimer: As stated in my previous article, messing around with your operating system like this is very dangerous. While I have made every effort to ensure the program works as described it is still possible that errors will occur. So make sure to back up your drive first. Use of this program is at your own risk.

Update [October 5, 2012]:
So despite my rigorous testing of the program it appears that it doesn’t actually work in the scenario in which I intended, the command prompt of the Windows Preinstall Environment (installer discs). Truth be told I never actually tested it in this scenario because it would have made development incredibly slow and complicated. Instead I tested using a duplicated data set on a separate pair of drives from my main system. Apparently that was a big mistake on my part since it would seem .NET based applications like mine don’t run at all in Windows PE. Bummer!

An aside: Why on earth would Microsoft develop the .NET framework and market it as the end-all-be-all of Windows’ platform development if they don’t include support in Windows PE? That seems like a huge gap in judgement.

Back to the real issue at hand. Since no C#/.NET programs will work I am rewriting the utility entirely in C++. So stay tuned!

Dissecting the Xyloband

A couple of nights ago I went to the Coldplay concert at the Hollywood Bowl. The show was amazing in every respect but what really grabbed my attention were the wristbands they handed out at the gate. These wristbands immediately lit up and began to flicker to the beat of the music, and coordinated with the rest of the lighting coming from the stage (to see it in action take a look at this video). It was truly impressive to say the least!

Being naturally curious as I am I started researching the little gizmo as soon as I got home. I found this great post on Hackaday that already began the discussion I was interested in; how does this thing work? That quickly led me to the interview with it’s creator, Jason Regler, and finally to the actual patent filed with the USPTO (of which I proceeded to read in detail).

My next step was to actually take the thing apart. I’m lucky to have two of these since my fiance Heather joined me on the occasion.

You can see from the picture above that it is powered by three 3V batteries. The two CR2016 batteries are stacked on the back side of the board while the CR2032 is attached to the front. The circuitry itself seems moderately complex with only two ICs and a number of resistors. The most visible IC is on the bottom left is an Si4313 RF receiver from Silicon Labs. This RF receiver runs in the 240-960 Mhz range. The second IC that is partially visible from underneath the battery clamp appears to be the C8051F980 Microcontroller (MCU), also from Silicon Labs.

And for good measure here is a shot of the back of the circuit board.

It’s a very cool gadget. As I dig a little deeper I hope to be able to turn this thing on at my own will, preferably without Charlie Brown’s help.

 

Help stop SOPA/PIPA

Several companies such as Google, Wikipedia and even some among my own industry (Razer), have declared today as Blackout Day in protest against the Stop Online Piracy Act (SOPA) and Protect IP Act (PIPA).

Online piracy is a serious concern and should be dealt with appropriately. Unfortunately these powerful pieces of legislation would empower government and anyone with enough lobbying influence (i.e. big corporations) the ability to ban and block users from accessing web sites and services deemed undesirable. This directly violates The First Amendment and should be stopped.

If you care about protecting civil rights and maintaining the freedom of the Internet please help join the fight.