Monday, October 27, 2014

To RAID or not to RAID...

For the last week I've been struggling to get my RAID-6 array back up and running. This is the second time I've had major issues with a RAID and I can tell you... it had me sweating bullets. I've sorta had a "come to Jesus" moment with my RAID array and I am looking to eliminate it with an alternate solution.

Here is my current RAID-6 setup:

This is a straightforward raid solution. 8 drives in a RAID-6 configuration where 2 drives worth of recovery are available   
What a RAID promotes is laziness in backup. You may have the best intentions but I would bet that after years of use, you will slack off of backups   
Then when you aren't paying attention it stops working and then the nervousness kicks in:   
1    How long has it been since the array has been backed up?
2    What will I lose if I can't get the array back up?
3    Oh crap, 2 drives are out of the array. If I lose one more…
4    And… a lot of curse words if it isn't coming back up.
1    The drives are one file system. No special considerations necessary for type of data
2    Can handle 2 drive failures before data is lost
1    If the Array fails (3+ drives are bad) all remaining drives are useless
2    Mounting a large raid file system takes 5-10 minutes
3    Adding a drive back into the array takes 1-2 days to complete
4    Regardless of the operation (read/write), all drives are hit to complete it
5    Backup drives can not be hot-swapped into the array. Data must be copied off them
6    The structure of the backup drives does not match the structure of the array
7    You need all the drives for your space right away even if half of the space is unused for years
      > NOTE: You can grow an array but it is unlikely you would do this
8    All drives need to be the same size in the array. Some can be larger, but the extra space is lost

Here is my new/proposed no-RAID setup:

This is my new solution. I will cover my rationale for exactly how I came to this conclusion later       
Components of the solution:       
1    Constantly changing data is dealt with separately with a glacial or cloud backup (documents)   
        ie: I will not be putting constantly changing data on this server.
        I will be keeping this data in a cloud or glacial back up
2    Once data is added to a drive, it never moves.   
3    When a drive is full, it is never touched again   
4    The drive that is currently being added to is backed up nightly via rsync   
5    When the current drive is full   
      a    The drive is never touched again
      b    The backup is taken out and a new backup is put in place
      c    A new "current" drive added into the enclosure
      d    A secondary backup is made outside the array of the latest full backup
      e    The secondary backup is stored in a different/safe location
1    Regardless of how many drives fail, the remaining drives are still useful   
2    Adding a backup drive into the enclosure immediately makes the system fully functional   
3    Only the drive with data is hit for read/write   
4    Likely, the latest drive is hit the most and the older drives remain inactive extending lifespan   
5    The backup drive exactly mirrors the file structure   
6    Backup is automatically handed by rsync   
7    There is more space available in the enclosure   
8    You only need new drives as you use up space   
9    New drives can grow in size as new drives become cheaper   
10    This approach encourages/enforces backups   
1    Data is partitioned by drive. This is not really a big deal   
2    This is slightly more manual in nature (rsync versus raid recovery)   
3    Finding a specific file may require looking onto multiple drives based on how organized you are   
4    A tiny amount of space will be unused on each drive when the latest file won't fit onto it   
5    This may require a few more drives.

Exactly what is stored on the file system?           
    You have 2 classes of data:       
    1    data that costantly changes   
          These are documents, a personal database and a personal source repository (< 1% of space)   
    2    data that never changes   
          audio, images, videos (> 99% of space)   
When to use a RAID?           
    If the contents of the file system are out of your control, by all means, use an array       
    If the majority of your data changes (large database), use a RAID       
    If you need to store something that exceeds the size of any one drive, use a RAID       
    Otherwise, I would suggest a RAID is overkill and does not provide the backup you think it will             
Unchanging data           
    Audio, Images and Videos never change.
    This is a very important consideration. If you are like me, that is 99.9% of your data.       
    No video will ever exceed 3TB. Once I put a video in place, it never moves.
    Why exactly would I want a RAID?       
Raid != Protection           
    A raid does not provide protection.
    If your drive monitor stops working or there is corruption in your array, you are screwed.       
    I say this this from practical experience.
    It has happened TWICE. Turn your back on a "smoothly" running array and it will fuck you.
    This is the LAST thing you want to hear when coming in from yard work:
    "dad, xbmc isn't working anymore for mom and me"       
How valuable is your data and your time?           
    Do you have a solid backup policy in place? This is your home btw. I highly doubt it.       
    Even if you have the data in some form somewhere, how long will it take to reconstruct it?
    How valuable is your time?       
    How much of the data can not be recovered?       
    How much time can you afford for your system to be offline while you recover data?       
    Who is affected by not having this data for weeks?       
Having a drive organization strategy           
    After many years of data hording, I think static data falls into these categories:       
    1    images   
    2    audio   
    3    video (you may choose to break this into a few subcategories)   
    4    software   
    5    pdfs   
    My intent is to have these same directories on each drive regardles if any data resides in it       
Additional thoughts           
    If you never expect a file to have the same name across drives you could write a nightly
    script to add symlinks       
        These symlinks would go from each drive onto your main linux OS drive
        Having the same specific directory structure on each drive would make this pretty darn simple   
    Give yourself a little wiggle room on each drive       
        If you use a video client like XBMC, things change over the years on what they support   
        You will likely have nfo files and/or custom dir structures for XBMC to recognize your files.   
        I have custom code I've written to rebuild these based on the files.

Thursday, September 11, 2014

Bungie's Destiny

I bought Destiny several days ago and started playing it last night.

Having built a game I have a whole new appreciation for just how much work went into making this thing. Cloth, detailed characters, gigantic maps, rpg style leveling, multi-player. There is a lot of stuff in this game. Of course, for half a billion dollars it better be good.

I chose to play an awoken hunter who specializes in precision distance shots.

I like the vibe of the game. It feels like an MMO Mass Effect RPG.

The map where you select missions looks a lot like the animated cartesian map from the animated Sinbad movie:
Destiny map

Sinbad's book of peace

I'm not sure how much time I will have to play it in the evenings but it will definitely become my morning treadmill workout game.

UPDATE - 09/30/2014

I've been exclusively playing Destiny since it came out. Hit a wall at 27. I've been there for about a week. I need an extremely rare material called Ascendant Shards to level up. I have 3 exotic weapons and are working on a bounty to get a 4th.

Wednesday, September 10, 2014

123D Catch

123D Catch by Autodesk

I just installed this application on my Nexus 4 phone. It allows you to take multiple photos and turn them into a 3D object. I'm going to try this out this week and report back how well it works.

You can download it here from the Autodesk store:

Monday, August 25, 2014

XBMC + Steam + DK2 + wireless XBOX Controller

The first thing I need to do as a VR enthusiast is get everything set up correctly in my living room. I can use the DK2 on my gaming rig but I want to try it out in a more casual setting.

After a bit of thought I have decided to build my own windows 8 steam box. I love the idea of Steam OS but not everything plays on it and I want a Steam console in my living room today, not several years from now.

Eventually Valve will release their steam controller which should complete the picture but this is a step in the right direction.

I have ordered the parts for the machine and will put it together in a few weeks. I went with part selections (mostly) from this website:

Some of the parts were out-dated but I like his picks. The idea is to have as silent a set up as possible. My current HTPC is not quiet and it is annoying.

When I start assembly I will update this page with photos and any snags I run into.


Update #1: 8/26/2014

I wanted to buy a simple, reliable RF remote with keyboard. I found the PERFECT remote:
The Boxee Box Remote for PC

Unfortunately, it is no longer available. wtf? This thing looks amazing, it has just what you need and it was RF. I haven't seen anything come close to it:

 I can not find this thing anywhere... and I looked a lot...

Monday, August 18, 2014

Switching to Enthusiast

VR development was taking up a large portion of my non-work time.

I produced a game in June (The Stacks) and it was taxing for myself and my family. Having to switch from UDK3 to UE4 was a huge relearning process and almost nothing I did carried over into the new engine. To rebuild the stacks, or any variant of it, would require tons of time and from what I've seen, IE4 is noticeably slower than UDK3. I just don't have the energy for it right now

I am going to switch to being an enthusiast for a while. I have set up the DK2 downstairs by the couch to see how well it works as a true consumer device.

As it stands, the DK2 has several issues:

1. Requiring a camera for the DK2's kinda sucks
I never thought about it until I set it up in the living room where my consoles are. Then it became obvious that it is worse to set up than the wii sensor bar or the kinect.

My consoles are to my right when I am sitting on the couch which is the best spot, but for most people, their consoles are probably across the room. This means you need to drag the camera and the Oculus cable across the living room to where you sit. Maybe the camera has a long range but I'm guessing it needs to be closer to you than your tv/screen.

The camera currently has 2 wires coming into it. This made for 3 wires coming from the computer. I tried to streamline the wires for the camera by twisty-tying them together but it is still a bit of a mess.

I think most consumers have had their fill of kinect and wii sensor peripherals and the DK2 camera falls into that category. At some point VR headsets will need to dispense with external head tracking.

2. Lack of major game developers porting games for VR is a big problem
In the beginning they had Valve. Team Fortress 2 and Half Life 2 for DK1 were great. Since then several of Valve's key VR folks left for Oculus. This seemed like great news at the time but now we don't have those people working for Valve to encourage game ports. The current DK2 port of Half Life 2 is not as good as the DK1 port. Darker areas look like 8 bit coloring. It looks rather nasty. Maybe if you'd never seen the DK1 version you'd love it but I'm not impressed.

3. The DK2's integration into windows is not pleasant
On both of the windows 7 computers I installed it on, the DK2 came up rotated in the wrong direction and I had to fight with it for 5 minutes on each one to get it to work properly. Multi-monitor support has never been simple but they need to address this problem in the final consumer product.

4. The DK2's colors are bad
I'm not sure what is going on but most things have a blue/red border around them. I find this to be worse than the screen door effect because it ruins immersion.

I would also like to see a headphone jack on the headset.

Using the DK2 reminds me of running a game on Windows 3.1. You have to fight with stuff a lot to make things go. I know it is a development kit so I expect issues but... the DK2 feels like a huge step backwards from the DK1 in terms of ease-of-use.

I think today's consumers expect it to just work when you plug it in. If Oculus can't figure that out and have a solid library of games available during their initial launch, it is going to be very rough start for them.

VR is the future but we are probably 5 years out from a product that works well in the home with a large library of games.

Friday, August 1, 2014

Stacks Hub Building Construction : Part II

Today I cut out the windows with the boolean feature of blender.

Man I hate that thing but sometimes it is the lesser of 2 evils. It made a total mess out of the mesh. It took me over an hour to fix faces and vertexes. Here is the result:

Added some window detail.

I added the chimney, ridges along the roof and headers to some of the windows:

I slid the chimney down to be along the the right exterior wall, removed the circular window from the right exterior wall, added a floor and interior wall for the top floor and welded the chimney onto the roof. I hid a few faces so you could see the interior.

I spent a few hours improving the arched windows to extend properly into the room. These arched windows are a total pain in the a$$.

I found this reference image for how to deal with them on the inside:

Here is the result:

Ignore the crappy texture mapping. That was me trying to figure out how to show inverted normals. It turns out there is a "backface culling" check box I somehow unchecked.

I fixed the fireplace interior, added a door to the right side for the second patio and added more detail to the lower windows.

I added a railing and window panes this morning

I added the upper windows, the lower "garden level" window and door headers. The interior of the first floor was messed up so I had to fix that too...

It's starting to feel like a real building. Still lots to add...

I improved the right entrance and added hand rails to the left entrance:

I started work on the right balcony. It is lower than the left one so I added a few step down onto it. Since I'm also working on a "perfect" version of the building I opted for wood planks versus haphazard sections of wood for the railing.

I added more to the right balcony.

I worked on the awning a bit this afternoon. It looks a bit weird because normal back-sides are not being rendered.

I added the table and barrels and took it into UE4:

Objects on table complete. Getting close...

I decided to enhance the detail of the light to make it a paper lantern:

Saturday, July 26, 2014

"The Stacks" new hub building construction

Humble beginnings but you gotta start somewhere...

I've extended the walls up and are preparing to join the vertexes above the doors.
The cube on the right is the doorway size. I need it to be consistent.

This will need to be exported to UE4 and viewed in the Rift constantly. The proportions of the building are tight. This will likely need to be deeper like a lego modular building. The original cg artist's primary goal was to replicate the look of the building. Heights of the doors were inconsistent among other things.

I wanted my steps to be realistic so I read up on how to build real stairs and created a spreadsheet calculator to build them. I realize there must be a plugin for this but it was a good exercise to understand how it is done.

Here are the finished entry steps:

With the spreadsheet it took about 2 minutes to build the top 2 steps.

I finished the steps, added an interior wall and floor for the left entrance. Now to add collision and bring it into UE4 for scale assessment.

Here is the collision. 13 collision meshes so far...

Well, getting this bugger into UE4 was a total suck fest. I learned a few things:
1. You need to create a UV map. I think you also need a light map. I did both at the same time. If you don't the object is black.
2. The new fbx binary export does not support multiple collision meshes. Totally annoying. You have to use "fbx 6.1 ascii".

I've got to figure out a better workflow. There seems to be a dozen new hurdles getting meshes from blender into UE4.

So, here is the building with collision in UE4:

I added it to the sample map and added a door. I like the proportion of the door but it is too short. I scaled it to fit the door opening and then scale the building along the X axis until the opening matched the door:

I now know the correct scaling for the openings is 1.140137 (they need to be that much wider).

Ignore the crappy look of the material. I have not properly UV mapped the mesh. I just wanted to get a sense of scale.

Here is what it looks like in the rift:

UE4 does not do a good job with a lot of things when you view it in stereo. Normals looks weird, reflections are bad, anti-aliasing is nasty, lens flare is annoying. I had to turn that stuff off and find a material with minimal reflections so it looked good.


I've decided the left portion of the building is a staircase. I've built 1 flight of stairs. I fought with the scale until it felt right. The doors are way too big but the space and the stairs feels good.

I wasn't happy with the building. It was just too tall.
1. I shortened the height
2. I removed a step from the staircase and shrunk the height of all steps.
3. I made the stair landings equal size

Once I'm happy with it I will add the rails. Something like this:

I figured out how to drop the height by 2 more steps. I added 2 more stories of steps to visualize the total building height. The height will be driven by the stairs.
first flight = 14 steps + 3 outside
second & third flights = 16 steps

Designing objects for VR is very hard because it has to have the perfect proportions inside the rift. In real life, 8 steps comes to eye level. I've got it doing the exact same thing in the rift so the steps are perfect. At the same time I have to make the building look like the artwork as much as possible. The building was never designed to be used so it is quite challenging.

Just having a little fun before I add the windows and top floors...

Add added the second floor and some windows. I tried my best to pick the best window proportions that matched the art and would work with the interior. I had to move the first floor window close to the door to work with the staircase. I flipped the staircase along the x axis to allow a window to work along the side.

Third floor basics in place:

I added the balcony, floors, stairwell walls and doorways for the 2nd and 3rd floors. That actually took a while...

I added the roof and created an arched window cut out. The cut out is positioned to cut the first window from the roof.

View from within UE4:

View from within Rift:

That's enough for tonight.