Genetec Reliability

[thewireguys 3]

Running a system that close to it's limits is asking for problems.

 

One thing I've run into is that when a RAID array is in a rebuild situation, disk throughput drops significantly.

 

Being at a point where doing something like adding one more client (especially mobile with transcoding), or other minor changes (frame rate, etc.) can take your whole system down is not a position you should be in.

 

 

What RAID cards are you using or recommend?

Started out with HighPoint a few years back, wouldn't even recommend them to an enemy now (arrays and volumes disappearing, and unrecoverable, without any logging messages at all, etc).

 

Been using Areca 1882ix series cards for a while now, and they have been working well.

 

Heard reasonably good things about Adaptec, too, but I haven't tried them yet.

 

I just inherited a system with a Adaptec RAID 52445 in a RAID 6 for the video storage. 2 drives are already starting to fail so I am interested to see how long it will take to rebuild and how the system is effected during the rebuild.

[ssmith10pn 0]

On the 10TB Avigilon/Dell PERK SATA servers which are RAID 5 it takes about 24 hours to rebuild under full load.

[SS360 0]

Ok, so from the looks of it there is something called SMT (Simultaneous Multi-threading Technology) as well as API (Application Programming Interface). These two things effect how much information a software can handle. Basically since we are building multiple core processors now, we need software that can divide and send information to multiple cores properly. I'm still researching this, however from the looks of it heavy software companies have been burdened by creating proper code when it comes to handling the ability to divide and send information to multiple processors at the same time. Now I'm still not sure if this is only on the operating system level or if it actually effects the individual software companies. If anybody has any information on this please share. Thanks!

[skomo 0]

I didn't know HD buffer had anything to do with it.

The recommended 256Mb limit is on an enterprise class server with 10,000 RPM SAS 6Gb/s Hard Drives in a Raid Array.

 

im yet to see 10k sas drives in avigilon servers and expansion units.

all the ones ive installed are 7.2k drives.

[ssmith10pn 0]

I didn't know HD buffer had anything to do with it.

The recommended 256Mb limit is on an enterprise class server with 10,000 RPM SAS 6Gb/s Hard Drives in a Raid Array.

 

im yet to see 10k sas drives in avigilon servers and expansion units.

all the ones ive installed are 7.2k drives.

 

 

Your correct. I was confused with the EMC√ NAS storage drives I use.

[thewireguys 3]

Indigovision clams they can support "200 cameras per NVR - 500Mbps throughput."

 

XSOAcAWn3ZE

[enduser 0]

I know this is an old thread, but I was searching through google as we are having problems with a Genetec installation, and the symptoms sound very similar. Cameras drop connections at random times, or flip over to the backup archiver for no obvious reason, connectivity to archiver roles is lost, gspot errors appear in the Genetec event logs. Try as we might, we cannot find an obvious pattern or cause.

 

We are running SC 5.2 SR6 and a mixed bag of IP cameras. We have dedicated directories, and our servers are bumping along the bottom on every indicator at the moment, nowhere near utilising a significant amount of network bandwidth, storage bandwidth, memory or CPU.

 

I would be very interested in hearing from the original poster, or anyone experiencing similar issues.

 

N

[ssmith10pn 0]

I know this is an old thread, but I was searching through google as we are having problems with a Genetec installation, and the symptoms sound very similar. Cameras drop connections at random times, or flip over to the backup archiver for no obvious reason, connectivity to archiver roles is lost, gspot errors appear in the Genetec event logs. Try as we might, we cannot find an obvious pattern or cause.

 

We are running SC 5.2 SR6 and a mixed bag of IP cameras. We have dedicated directories, and our servers are bumping along the bottom on every indicator at the moment, nowhere near utilising a significant amount of network bandwidth, storage bandwidth, memory or CPU.

 

I would be very interested in hearing from the original poster, or anyone experiencing similar issues.

 

N

 

Where are you located?

[devsec 0]

You should really contact their costumer support as they are more likely to be able to help you resolve your issues.

 

That said, can you provide more information about your problems? Did you start experiencing these issues after an upgrade or is this a new installation? Are the cameras that are dropping from a specific manufacturer or are they all dropping? Have you tried analyzing network traffic to see if the problems could be network related? What else have you tried?

[enduser 0]

Hi,

 

we have had multiple sessions with out local integrator on site and Genetec engineers remotely accessing the system from Canada to investigate the issues, and we are making little progress unfortunately.

 

The installation is about a year old, and has had the problem all along, initially on SR2, and continuing since we upgraded to SR6. SR6 did solve one issue, a memory leak in WMIPRVSE, which reached an internal memory cap, then died and took the archiver with it. WMIPRVSE is actually a windows module, not a Genetec one, but possibly SR6 is using it differently.

 

I've exported months worth of dropout logs into a database so that I could try to spot a pattern, I've looked for patterns in archiver allocation, camera type, vlan, network switch, date and time, pretty much everything I could think of. We have also wiresharked the archivers through a number of dropouts and sent these to Genetec, as well as examining them myself and by the network engineer who designed the local network.

 

The only apparent patterns I can spot are:

 

1) If I reboot the archivers and directory servers, generally things seem stable for about 2 weeks, then the issues start to reoccur.

2) One particular camera type, Oncam, often fail at 5 seconds past the minute. Sometimes they recover instantly on the same archiver, sometimes they flip to the backup archiver, then flip back 60 seconds layer. In the second case, we see a ping from archiver to camera right on the minute, with no response from the camera. In that case the server sends an rtsp teardown to the camera, which then fails onto the backup server. on the next minute there is another ping, a response is received from the camera and the primary archiver re-initiates the rtsp session. In the first case, there is no obvious cause the wireshark output.

[ssmith10pn 0]

Was this system an Omnicast 4.6 to 4.8 system Migrated to Security Center? Or has it been 5.2 from the start?

 

How big is the system?

Camera Quantity

Number of Archivers

Dedicated Directory Server?

Any Federations?

 

Please give some more details about the network.

 

Have you considered going to SR8?

[devsec 0]

If a camera fails to respond to a ping request, then it will be assumed to be offline, even if it was still streaming. I see that Oncam was not integrated with onvif although they are onvif profile S compliant. Perhaps they weren't compliant back when they were integrated. Can you try adding these cameras with the onvif manufacturer instead of Oncam? I know that you will lose the dewarping feature but with some config file wizardry you could get that back. If this proves successful, you could request that Genetec integrates Oncam with onvif instead and they could provide you a temporary fix that includes dewarping while they are working on an official fix.

 

Are there any other issues with the servers themselves or other types of cameras?

[enduser 0]

Hi,

 

> ssmith10pn

 

The SR2 installation was our first Genetec deployment, so this is a clean build, not a migration, cameras, clients, servers and storage are all new, as is the supporting network design.

 

There are no federations.

 

The system currently comprises 2 dedicated directory servers (active & backup) and six archiver servers - each archive server hosts one primary and one backup archiver role. All the servers correspond to the Genetec "Large" specification, so 300Mb/sec capacity. Servers are split across two locations for resilience, each archiver has 2 8Gb FC HBAs connected to a pair of FC switches (Multipath IO enabled) then a stack of SAN storage, and each server uses a single Gig ethernet port for cctv feeds. The SANs are not interconnected.

 

 

There are only about 100 cameras on line at the moment, a variety of Axis, some oncams, a very small number of areconts. All these are IP cams, and all suffer from sporadic failovers, but only the the oncams exhibit the "5 seconds past" thing. More cameras are due to be integrated as building works progress, currently we are way below capacity, our busiest archiver is only peaking at 150Mb/Sec, the quietest is running about 20Mb/sec. On CPU the servers rarely exceed 15% and have plenty of RAM free, and the underlying OS is 64 bit so no 4GB memory limitations. SAN performance figures are also minimal (in a good way) at this time, no sign of any write queues or high CPU utilisation.

 

The server tier is built for more cameras than currently connected, more cameras are being rolled out as building works progress - for the current camera usage we are very over-specified with servers and storage. Also the server tier is specified to support the whole camera set from one server room if the other goes away for any reason, so under normal utilisation we should not be stressing our servers or storage.

 

The quiet servers (or rather their cameras) seem to fail just as often as the busier ones - we left them "unbalanced" to see if loading was significant in terms of camera dropouts, but it doesn't appear to be.

 

The underlying network is MPLS based, built on CISCO kit, core/distribution/access layered, and CCTV sits in a private VPN subdivided into separate client, server and camera VLANS. Cameras and clients are divided into two VLANS each in order to balance load across the inter-switch uplinkslinks. Only essential traffic is allowed in & out of the VPN (AD authentication, AV updates, that sort of thing) Cameras and servers use fixed IP addresses, with clients on DHCP, and it is IPV4 only. Network engineer reports plenty of spare capacity on the network, and no obvious issues in the switch logs.

 

AntiVirus is installed on the servers, but an exclusion prevents scanning of the Genetec binaries and all the drives used for video storage - I have checked and as far as I can tell this is working as planned, Resource Monitor shows no access to these locations from the AV service.

 

I've not seen anything in SR8 release notes that looks like it might address the issues, but we are considering it regardless.

 

> devsec

 

I won't pretend I understand the implications of your integration suggestion, but will talk it through with somebody who does I'm more from an IT apps and infrastructure background, this is my first foray into CCTV. Losing the dewarping capability would be a no-no, the users like that a lot.

 

To answer your questions:

 

1) All the camera types have issues, but the Oncam does have this specific "5 seconds past" one all to itself.

2) Sometimes we see gspot errors in the "Genetec" logs in windows event viewer, and sometimes "connection to archiver role lost" messages. These can correspond to camera dropouts, but often dropouts happen without these messages appearing.

 

 

One thing I would say, is that the actual cctv users seem very happy with the system in general. Some dropouts cause a short loss of recorded footage, but users are not complaining of a loss of live view. Obviously that short outage would be critical if it occurred during some sort of incident though, that's my main concern.

 

Thanks for your interest, any suggestions or queries are greatly appreciated.

 

Neil

[devsec 0]

If cameras other than Oncam also drop, you can disregard my comment about onvif as it probably won't help you. There seems to be a general issue with your installation and finding the cause won't be easy because it all happens "sometimes". I'm not quite sure I fully understand what is happening on your system. When you say that the cameras dropout, do you mean that they become red in Security Center? After the 2 weeks period, how often would you say that you experience the dropouts, the failovers, the gspot events and the "connection to archiver role lost" messages? You say that your users are not complaining about live video loss but you are missing some recordings, I assume this means that the cameras are doing multicast? How long are the missing spot in your archives?

 

I would highly advise that you put pressure on the Genetec costumer support until your issue is really taken care of.

[enduser 0]

Fingers crossed we have overcome the dropout issues we were seeing. Nothing was "obviously" fixed, but the network team found some errors on the access layer switch used by one of the server stacks, relating to the distribution switch uplinks. These had no obvious cause, they tested the fibre, which came up good, and re-seated the SFPs, and that was all they did (or all they are admitting to )

 

No dropouts for almost six weeks now.

[ssmith10pn 0]

Sorry,

I must have missed this.

Several of our sites have issues with Axis cameras crashing their Http server.

The camera will ping but no video of web page.

the easy fix is an ftp reboot

 

cd\

ftp 10.x.x.x

User name

password

then type "quote site reboot" (without the quote marks)

 

this prevents you from having to bump the camera at the switch.

 

I have noticed on Axis cameras if you force the connection to TCP they run a lot more stable.

 

If the camera looses connection with the archiver it will crash trying to "phone home"

When ever a fiber link drops out for what ever reason we find a ton of of cameras that are locked up when the connection is restored.

In a campus environment outages are going to happen. it's the nature of the beast.

[genUser 0]

Fingers crossed we have overcome the dropout issues we were seeing. Nothing was "obviously" fixed, but the network team found some errors on the access layer switch used by one of the server stacks, relating to the distribution switch uplinks. These had no obvious cause, they tested the fibre, which came up good, and re-seated the SFPs, and that was all they did (or all they are admitting to )

 

No dropouts for almost six weeks now.

 

We've been testing Genetec Security Center 5.2 SR9 to see whether we want to adopt Genetec's video software. Our site has some 250 cameras from a variety of vendors.

 

We have seen the same issue of Genetec silently stopping and restarting recording on a random basis. We're nowhere near capacity as we've seen the issue on one test server that is handling exactly one camera as well as another that only has 12 test cameras attached. I came to this forum to see if any other sites have had the issue.

 

Could you elaborate on the nature of the errors the networking group found? Our network group swears up and down everything is fine but Genetec's logs indicate some recording transactions are randomly taking up to two seconds instead of milliseconds. The networking group says their software isn't showing any glitches that would account for the long duration transactions.

 

We've also had problems with the Access Managers losing connectivity with the door locks. That issue was resolved when we scheduled an off-hours, weekly reboot. That worked for the locks but it's not an option to lose video on a scheduled basis.

[ssmith10pn 0]

Genetec is the better solution on large Enterprise systems.

Genetec Certification is a Must. This gives you access to their private forums and a ton of Documentation.

 

They support so many legacy cameras it's almost a break fix with every service release.

SMA is required also.

 

For a 250 camera system I would go with Avigilon if all the cameras were supported.

 

For a 1000+ camera system Genetec is a no brainer.

[Arstron 0]

Anyone know if there is a 64 bit version of Genetec Security center for the client/workstation? We are running windows 7 64 bit with 16 gb of ram on our client computer, but memory usage is not going beyond 4gigs. We got out of memory errors on random cameras, the software runs pretty slow when doing task such as assigning badges or switching tabs.

[enduser 0]

Apologies for the late response, if there was a "new post" notification I missed it.

 

Genuser, the errors found related to the UDLD protocol, which I believe is related to routing management for the LAN, but I'm not going to pretend I understand the ins & outs of it.

[genUser 0]

Fingers crossed we have overcome the dropout issues we were seeing. Nothing was "obviously" fixed, but the network team found some errors on the access layer switch used by one of the server stacks, relating to the distribution switch uplinks. These had no obvious cause, they tested the fibre, which came up good, and re-seated the SFPs, and that was all they did (or all they are admitting to )

 

No dropouts for almost six weeks now.

...

We have seen the same issue of Genetec silently stopping and restarting recording on a random basis. We're nowhere near capacity as we've seen the issue on one test server that is handling exactly one camera as well as another that only has 12 test cameras attached. I came to this forum to see if any other sites have had the issue.

...

...

 

We've resolved the issue and it wasn't Genetec's fault. Genetec relies on the time being accurate at powerup and unfortunately, our server's bios clock was wrong. It got fixed by an NTP call early in the power up sequence but Genetec had already started up which meant it had the wrong time.

 

To spot the issue, we filtered the Windows System event log to focus just on startup and powerdown sequences. The raw, unsorted listing showed that startup and powerdown events were mismatched in time.

[enduser 0]

I'll look into that, but to be honest I would expect an NTP call to feature much earlier in the boot-up sequence than Genetec starting.

 

That said, there is a registry setting:

 

HKLM\Software\Policies\Microsoft\Windows NT\CurrentVersion\Winlogon\SyncForegroundPolicy

 

I have had trouble with this before, it may be worth setting this to 1 to make windows and apps wait for the networking stack to sort itself out before letting apps run.

[Left Turn Larry 0]

I am the integrator. I am running Genetec 5.1 with service pack 8. It just seems that Genetecs software architecture was designed for analog cameras. As an example Genetec has a software limitation of processing 300mbps when running a stand alone archiver, but if you want to run the archiver and directory on the same machine it has a 200mbps limitation. This limitation was causing me to fail on pulling video from the archiver even though my hardware could handle the load. The software would fail the video extraction because it was causing over the limit network processing to occur. So since I am running the directory and archiver on the same server any time my gigabit network would spike over 20% the video extraction would fail. This limitation is all processing not just the bandwidth from the camera to the archiver, but from the archiver to the guard viewing client as well as to the video extraction no matter what machine it was being pulled off of.

 

Another example is the Security Desk client has a 5gb cache limit when viewing stored video directly out of tiles. The software will cache the video so it can be scrolled through faster which makes sense, however since I am running 5 megapixel Arecont cameras (which suck for bandwidth/storage) So when the customer reaches that 5gb limit of cached video it just stops allowing any more video to be viewed and after my rounds with Genetec the only way to clear it as of now is to close the program and re-open it.

 

The system I am referring to only has 24 physical cameras but due to the 180 cameras I have 33 video feeds.

 

I have an Exacq system running 55 Arecont 5 megapixel cameras and have no problems with it. So I was just seeing if anybody else is experiencing these issues with Genetec.

 

 

I realize this is an older post, however I have several customers still using SC 5.1 with one server and one client. I have been fighting with these genetec servers for a long time now. I have several, separate locations with lagging issues. Constant buffering video, running slow. One thing I was told by genetec was to turn video caching off, hasn't fixed anything. One of my system runs 95 axis cameras, again one server, one client. I am beginning to wondering if we are setting them up correctly. The instructions come from the client/genetec/IT dept. I don't understand why a client pulls up video to review and it buffers for 30secs before you can review video. Where is the bottleneck? Client machine or server?

 

This post helped me alot as for getting some kind of insight into how this software functions. With the cost of this software and the robust(ness) of the software, I don't think i should have any issues with lagging video

[ssmith10pn 0]

Get off 5.1 ASAP.

It was terrible!

5.4 much better!

[ssmith10pn 0]

The little system I am responsible for.