ftdi_read loses bytes (when latency is low)

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

ftdi_read loses bytes (when latency is low)

Hendrik
Hi,

I am currently porting our software from Windows (using FTDI's own
driver) to Linux (using libftdi). Thus far it works great but I am now
on a specific case in the program that needs to read chunks of data from
our device.

The commands between the host (now a PC, later an embedded machine like
a RPi) and our device are all 'request' -> 'reply' based. This specific
case requests 522 bytes of data from the device that are sent over a
500.000 bps link. Note that there is no flow control on the link but the
incoming and outgoing packets have a fixed / predictable size and a
reply can only follow a request. The program flow is linear:

'ftdi_write' the request
while (not all 522 bytes received)
     'ftdi_read' the reply

When latency is 16ms or something, this guarantees the reply to be
created by our device before the latency timer expires. In that case the
problem does not occur (or not as frequent to spot it anyway).

When latency is at 1 ms, our device is not ready in time with the reply,
which means that it takes about 6 iterations of expired latency timers
and ftdi_read calls that return with 0 bytes. After that the complete
chunk of 522 bytes comes in.That is ok for me BUT every once and a while
not the complete chunk is received and I receive less that the expected
522 bytes. This happens more often if the computer is busy. I also never
seem to hit the case that ftdi_read does not give me the expected bytes
(or nothing), so the while loop for an incomplete read (except the
0-case) seems redundant.

I think this happens because the host does not poll the USB bus often
enough and therefore loses data that is sent by the device but that is
just a guess because:

* The read chunk size is at 4096 and therefore larger than expected 522
bytes
* If the latency timer is higher, the problem does not seem to occur (so
at what moment can we lose bytes, where are bytes stored when the
latency timer has not expired yet and can we lose bytes just as well?)
* The internal FTDI buffer is only 128 bytes (for sending) anyway so how
can I even receive chunks of 522 bytes at one ftdi_read command
* The first couple of ftdi_read calls return in about 1 ms (latency)
with 0 bytes revceived but THEN I get one single call (that takes
 >10ms!) that return with 'less than expected'. If I am inside the
ftdi_read during that time, how can I ever lose these bytes if there
were not there the call before? Note that the same applies if a call
succeeds: a couple of 1 ms call, then a ~12 ms call with the complete
packet.

Can anyone explain this to me (and maybe hand me some insights or a
solution :))?

Some more info:

* I checked with usbdump and it indeed seems that not all bytes are
actually transferred over the USB bus
* I know that the FTDI driver in windows seems to be polling all the
time (if on 1 ms latency it sucks away 'kernel CPU' especially in our
case with more than 1 FTDI connected and the port open) but the
behaviour I see here does not happen (or VERY infrequently in Windows)
* I am on kernel '3.16.3-031603-lowlatency', could the 'lowlatency' part
be interfering?

Thanks in advance!

Regards,
Hendrik





--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]  

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ftdi_read loses bytes (when latency is low)

Uwe Bonnes
>>>>> "Hendrik" == Hendrik  <[hidden email]> writes:

    Hendrik> Hi, I am currently porting our software from Windows (using
    Hendrik> FTDI's own driver) to Linux (using libftdi). Thus far it works
    Hendrik> great but I am now on a specific case in the program that needs
    Hendrik> to read chunks of data from our device.

    Hendrik> The commands between the host (now a PC, later an embedded
...

No answer to your questions but another questions from my side:

- Why are you useing libftdi and not the kernel driver ftdi_sio?

- Do you have some usbdump snipplet where the problem happens?

- Can the USB chipset be involved here?

Bye
--
Uwe Bonnes                [hidden email]

Institut fuer Kernphysik  Schlossgartenstrasse 9  64289 Darmstadt
--------- Tel. 06151 162516 -------- Fax. 06151 164321 ----------

--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]  

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ftdi_read loses bytes (when latency is low)

Uwe Bonnes
In reply to this post by Hendrik
>>>>> "Hendrik" == Hendrik  <[hidden email]> writes:

...
    Hendrik> * I checked with usbdump and it indeed seems that not all bytes
    Hendrik> are actually transferred over the USB bus * I know that the
    Hendrik> FTDI driver in windows seems to be polling all the time (if on
    Hendrik> 1 ms latency it sucks away 'kernel CPU' especially in our case
    Hendrik> with more than 1 FTDI connected and the port open) but the
    Hendrik> behaviour I see here does not happen (or VERY infrequently in
    Hendrik> Windows) * I am on kernel '3.16.3-031603-lowlatency', could the
    Hendrik> 'lowlatency' part be interfering?

Even another question:
Did you try asynchronous transfers (ftdi_read_data_submit)?
--
Uwe Bonnes                [hidden email]

Institut fuer Kernphysik  Schlossgartenstrasse 9  64289 Darmstadt
--------- Tel. 06151 162516 -------- Fax. 06151 164321 ----------

--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]  

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ftdi_read loses bytes (when latency is low)

Hendrik
In reply to this post by Uwe Bonnes
Hello Uwe,

Hendrik> Hi, I am currently porting our software from Windows (using
Hendrik> FTDI's own driver) to Linux (using libftdi). Thus far it works
Hendrik> great but I am now on a specific case in the program that needs
Hendrik> to read chunks of data from our device.
Hendrik> The commands between the host (now a PC, later an embedded ...
 > No answer to your questions but another questions from my side:
 >- Why are you useing libftdi and not the kernel driver ftdi_sio?
I am using the libftdi ftdi_read function and have unloaded the ftdi_sio
kernel module. I also updated to libftdi 1.2 today but the same problem
occurred on 1.1 (I updated today because I wanted to know if the problem
went away with this version but it didn't).

 >- Do you have some usbdump snipplet where the problem happens?
Not at home but I will send this tomorrow when back at work

 >- Can the USB chipset be involved here?
Maybe :), the motherboard is an Asus M4A785-M (I think) with AMD710
southbridge (Support up to 14 USB ports (12 USB 2.0 and 2 USB 1.1))

 >- Did you try asynchronous transfers (ftdi_read_data_submit)?
No, I try/tried to stay away from async transfers as the whole protocol
is request->reply based which makes it easier to program (and easier to
read what's going on).

I looked through the libftdi sources but can you tell me what the
latency timer actually does (in combination with the read chunk size)? I
know its function and that a read command will return when the timer has
expired and the chunk was not competely read, but where is this
information stored? The ftdI_read_data function don't seem to use the
timer so it must be hardware based, but if the FTDI buffer itself is not
as large as the 4KB chunk, where is this data buffered then in
combination with the timer?

I will post some more debugging info tomorrow.

Regards,
Hendrik

--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]  

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ftdi_read loses bytes (when latency is low)

Uwe Bonnes
>>>>> "Hendrik" == Hendrik  <[hidden email]> writes:

    Hendrik> Hello Uwe, Hi, I am currently porting our software from Windows
    Hendrik> (using FTDI's own driver) to Linux (using libftdi). Thus far it
    Hendrik> works great but I am now on a specific case in the program that
    Hendrik> needs to read chunks of data from our device.  The commands
    Hendrik> between the host (now a PC, later an embedded ...
    >> No answer to your questions but another questions from my side: - Why
    >> are you useing libftdi and not the kernel driver ftdi_sio?
    Hendrik> I am using the libftdi ftdi_read function and have unloaded the
    Hendrik> ftdi_sio kernel module. I also updated to libftdi 1.2 today but
    Hendrik> the same problem occurred on 1.1 (I updated today because I
    Hendrik> wanted to know if the problem went away with this version but
    Hendrik> it didn't).

This doesn't tell _why_ you don't use the kernel driver...
I think the kernel driver is better in such corner cases.

    >> - Did you try asynchronous transfers (ftdi_read_data_submit)?
    Hendrik> No, I try/tried to stay away from async transfers as the whole
    Hendrik> protocol is request->reply based which makes it easier to
    Hendrik> program (and easier to read what's going on).

The code flow isn't that different.
Without aync transfer:
'ftdi_write' the request
while (not all 522 bytes received)
     'ftdi_read' the reply

With async transfer:
'ftdi_read_data_submit' setup the read request
'ftdi_write' the request
while(read transfer not complete) wait;

    Hendrik> I looked through the libftdi sources but can you tell me what
    Hendrik> the latency timer actually does (in combination with the read
    Hendrik> chunk size)? I know its function and that a read command will
    Hendrik> return when the timer has expired and the chunk was not
    Hendrik> competely read, but where is this information stored? The
    Hendrik> ftdI_read_data function don't seem to use the timer so it must
    Hendrik> be hardware based, but if the FTDI buffer itself is not as
    Hendrik> large as the 4KB chunk, where is this data buffered then in
    Hendrik> combination with the timer?

    Hendrik> I will post some more debugging info tomorrow.

Either way, the libftdi problem needs to be solved, so thanks for your
patience.

Bye
--
Uwe Bonnes                [hidden email]

Institut fuer Kernphysik  Schlossgartenstrasse 9  64289 Darmstadt
--------- Tel. 06151 162516 -------- Fax. 06151 164321 ----------

--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]  

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ftdi_read loses bytes (when latency is low)

Hendrik
In reply to this post by Uwe Bonnes
Hello Uwe,

I made a USB dump of the situation. My device is instructed to report
back the 522 bytes packet that contains a 512 byte payload with the
pattern 00 01 02 ... fd fe fe 00 01 ... fd fe ff. You can see that at
the start of the dump it works fine, but then after a while an
incomplete packet is received which is missing data in the middle. These
kind of patterns (some ok packets, then an incomplete packet) repeat
over and over.

The pastebin of this you can find here:
http://pastebin.com/raw.php?i=KqvwqGGf and you can also find the problem
part in a separate pastebin at http://pastebin.com/raw.php?i=3Vwk2NAs 
which is cleaned up a bit to show that there is indeed data missing.

When the latency timer is set to 16ms it looks like this
http://pastebin.com/raw.php?i=1aSYzbXu and there are no lost bytes
anymore so it seems like there is some interference of  the latency
timer expiring and the data being received by USB.

Or that somehow the timer expires twice while a data transfer is already
in progress.

The
http://www.ftdichip.com/Documents/AppNotes/AN232B-04_DataLatencyFlow.pdf 
speaks about these corner cases and there are also some more reports on
this mailing list about lost data but those cases are very specific in
the way that there is either 'unlimited' data coming in (not bounded to
a certain size but an ongoing stream of data) or the fact that there is
full duplex communication going on (so USB writes interfere with the USB
reads). In my case I am only waiting on a packet (even waiting on USB
traffic at all) to arrive which in turn does arrive but is broken
somehow..?!

[edit, another post from Uwe came in]
 > This doesn't tell _why_ you don't use the kernel driver...

Well, I don't really know either ;). I have been using the Windows FTDI
driver with their lower level API because Windows serial ports suck. The
API allows me to do some calls that buffer the data the way I want and
we can also address the port by their USB specific information (e.g. the
serial number of the device instead of just 'COM<something>' (which is
cumbersome especially if there are many devices connected). Also the API
allows me to do a 'read' with a 'number of bytes' parameter that ONLY
returns if either that number of bytes are present OR the given timeout
expires. The standard serial port implementations, given a number of
bytes and a timeout ALSO return when less that the number of requested
bytes are present (as long as it is not 0 bytes or the timeout expired),
so I had to do the 'reassembly' of packets myself (using C# this was not
nearly as efficient as the FTDI driver just giving me what I wanted OR a
timeout).

So, thus far I haven't really looked at FTDI-SIO as I expected it to be
a 'serial port driver' instead of being a lower level API to get what I
want. I might give it a try now though :). I also haven't tried the
closed source driver from FTDI themselves, but mostly because it is,
well, closed source. I have experienced a lot of 'weirdness' (but never
lost bytes :)) using their Windows counterpart.

I will also implement the async version and see what it does and let you
know.

[edit 2]
I now see that as soon as the transfer goes wrong, the 2 status bytes
that are transferred over USB (and don't show up in the packet data)
start with 0162 instead of 0160 in the cases it goes OK. It remains 0160
until the next data packet is coming in...

Thanks so far and regards,
Hendrik

--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]  

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ftdi_read loses bytes (when latency is low)

Uwe Bonnes
Hello Hendrik,

more reasoning:
- What exact device are you using? How large is the receive buffer?
- Do you use any signalling between the FTDI and your serial device?

With 500 kBaud you receive about 50 bytes per ms. With a small receive
buffer and no RTS/CTS signalling on the FT232R with its 128 byte buffer,
overrun will happen in 3 ms. Your programm will wait forever for the lost
bytes.

The libftdi programm runs as a user program and other programms may keep your
from the CPU quite for some time. Realtime kernels, as I understand it, only
guarantee latency for kernel drivers, but not for a user program. So again,
using the kernel driver may help. And if things go wrong with the kernel
driver, it's a POOP (problem of other people) :-)

- Is the Device directly plugged to the PC or via a hub.  At least some time
ago, a USB 2 full speed (vs USB2 high speed) device directly connected to
the PC was only seviced from the linux PC in 1 ms steps. With an hub in between,
USB micro frames were used and things happened in 125 us steps. This could
also save you from overruns.

- Is it a real FTDI device or some fake rebuild?
In the latter case, you may hunt some FTDI-unrelated issues.

>>>>> "Hendrik" == Hendrik  <[hidden email]> writes:

...

    Hendrik> [edit 2] I now see that as soon as the transfer goes wrong, the
    Hendrik> 2 status bytes that are transferred over USB (and don't show up
    Hendrik> in the packet data) start with 0162 instead of 0160 in the
    Hendrik> cases it goes OK. It remains 0160 until the next data packet is
    Hendrik> coming in...

I don't see this behaviour in th pastbin logs...

Bye
--
Uwe Bonnes                [hidden email]

Institut fuer Kernphysik  Schlossgartenstrasse 9  64289 Darmstadt
--------- Tel. 06151 162516 -------- Fax. 06151 164321 ----------

--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]  

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ftdi_read loses bytes (when latency is low)

Hendrik
Hello Uwe,

 > - What exact device are you using?

I am using the FT232R (actually the FT232RL in 28SSOP casing).

 > How large is the receive buffer?

I mentioned 128 bytes before, but it depends on what you think TX and RX
means ;). The datasheet says:

 >> FIFO RX Buffer (128 bytes). Data sent from the USB host controller
to the UART via the USB data OUT
 >> endpoint is stored in the FIFO RX (receive) buffer. Data is removed
from the buffer to the UART transmit
 >> register under control of the UART FIFO controller. (Rx relative to
the USB interface).
 >> FIFO TX Buffer (256 bytes). Data from the UART receive register is
stored in the TX buffer. The USB
 >> host controller removes data from the FIFO TX Buffer by sending a
USB request for data from the device
 >> data IN endpoint. (Tx relative to the USB interface).

So the buffer that the microcontroller sends its data to is 256 bytes.
That gives a bit more margin for 'not servicing' but there is not enough
room for the complete packet to stay there a while.

 >  Do you use any signalling between the FTDI and your serial device?

No there is no handshaking whatsoever, as soon as my device has the
requested data complete it will send it in one go out of its serial port
(into the FTDI device). In Windows however, the device and this
configuration has run fine for about 5 years. We even have setups
connecting about 35 of these devices to USB hubs simultaneously and it
'just worked' (with some quirks from the Windows FTDI driver though).
But I do think that the Windows drivers are running their own very low
level processes that keep on polling the USB devices because the load on
the computers increase dramatically when using more of these FTDI
devices simultaneously (only opening the port already gives a lot of
stress on the CPU) especially with the latency set to a low value).

 > With 500 kBaud you receive about 50 bytes per ms. With a small
receive buffer and no RTS/CTS signalling
 > on the FT232R with its 128 byte buffer, overrun will happen in 3 ms.
Your programm will wait forever for the lost bytes.
 > The libftdi programm runs as a user program and other programms may
keep your from the CPU quite for some time.

That is indeed what I suspect is happening, an overrun occurs just at a
timeslot that my program is not servicing the USB read request. I don't
know the internals of libusb so I thought there would be/could be some
logic inside to handle this but if it is all userspace that possibly is
not the case. But WHY only with a low latency and never with a latency
that is high(er than the time it takes for the microcontroller to
assemble and send the complete packet).

 > Is the Device directly plugged to the PC or via a hub.

The device is connected to a USB HUB (as we need to connect many of
these to an embedded system in later stages of development). I will test
though if the problems goes away when connected to USB directly. I also
tried the async method by the way but that does not help. It makes it
'worse' as the 'ftdi_transfer_data_done (tc)' function never returns (it
does not meet up with the requested number of bytes, I found a patch by
you for adding a timeout to the 'ftdi_read_data_submit' call but it is
not in the mainline branch?).

 > Is it a real FTDI device or some fake rebuild?

Our PCBs are manufactured by a Dutch company. They have a reputation to
keep up and I don't think they use Chinese remakes of IC's. Also; in
Windows this problem is not observed at all.

[The 0162 USB messages]
 > I don't see this behaviour in th pastbin logs...

They are there but a bit hidden: http://pastebin.com/raw.php?i=3Vwk2NAs 
You can see that the packet with size 327 comes in with 0162 and the
following packets show 0160 again.

In this post you find a larger log of the program running for some time
(constantly requesting the data packet with a latency timer at 1ms, the
chunk size at 4096 and my 'read' function having a timeout of 1000 ms.
My read function is a wrapper that calls ftdi_read_data for as long as I
did not receive the expected total number of bytes until my own time out
expired).

http://pastebin.com/raw.php?i=C9qb7axf

I am not completely sure of the status bytes and their bit order (B7 ...
B0 or is it B0 ... B7). We receive 62 instead of 60 for the second byte
on error, this COULD be the 'overrun' bit (B1) which makes sense
(however it is not directly connected to that specific 64 byte packet,
the overrun is probably observed in the data after receiving about 256
correct bytes (the RX buffer size)). The first byte however indicates 01
but that a bit (B0) that is documented to be 'must be 0' which it isn't).

So, to summarize: it is probably a problem that hits me 'by design'. I
should not try to send buffers larger than the RX buffer of the FTDI
chip (without using some kind of flow control) under Linux, at least not
in user mode and with a low latency..?

Still, I don't get why the problem does NOT appear at all when I select
a latency that is (way) larger than the time it takes for my device to
respond (approx 4 or 5ms for packet building and then about 10ms (522
bytes @ 500kbps) for the actual serial transmission to the FTDI device).
What is really different in those cases? Where is the data stored when
the latency timer is higher? And why does the latency timer seem to
interfere with the ongoing transmission. Does it go off twice but isn't
it serviced twice, hence the data loss? But why do I miss a random
amount of data and not a 64 byte frame then? Questions, questions,
questions... :(

Regards,
Hendrik

--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]  

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ftdi_read loses bytes (when latency is low)

Uwe Bonnes
>>>>> "Hendrik" == Hendrik  <[hidden email]> writes:

...

    Hendrik> I am not completely sure of the status bytes and their bit
    Hendrik> order (B7 ...  B0 or is it B0 ... B7). We receive 62 instead of
    Hendrik> 60 for the second byte on error, this COULD be the 'overrun'
    Hendrik> bit (B1) which makes sense (however it is not directly
    Hendrik> connected to that specific 64 byte packet, the overrun is
    Hendrik> probably observed in the data after receiving about 256 correct
    Hendrik> bytes (the RX buffer size)). The first byte however indicates
    Hendrik> 01 but that a bit (B0) that is documented to be 'must be 0'
    Hendrik> which it isn't).

From e.g. ftdi.c
   2048 /**
   2049     Poll modem status information
...
   2071     Layout of the second byte:
   2072     - B0       Data ready (DR)
   2073     - B1       Overrun error (OE)
   2074     - B2       Parity error (PE)

So this is clearly an overrun (OE). Either handle that overrun in your code
and firmware, or use an asynchronous transfer, where libusb cares for you to
poll the FTDI in time, or use the kernel driver, where the kernel is in
charge to poll for you.

Hope this helps
--
Uwe Bonnes                [hidden email]

Institut fuer Kernphysik  Schlossgartenstrasse 9  64289 Darmstadt
--------- Tel. 06151 162516 -------- Fax. 06151 164321 ----------

--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]  

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ftdi_read loses bytes (when latency is low)

Anders Larsen
In reply to this post by Hendrik
Hello Hendrik,

On 2014-12-02 11:42, Hendrik wrote:
> we can also address the port by their USB specific information (e.g.  
> the serial number of the device instead of just 'COM<something>'  
> (which is cumbersome especially if there are many devices connected).

when you use the standard driver 'ftdi_sio', 'udev' will automatically  
create symlinks below /dev/serial/;
in the directory /dev/serial/by-path/ you'll find symlinks named after  
the topology ("where is the device plugged in"), and in  
/dev/serial/by-id/ you'll find symlinks named after the product name  
and serial number ("which device is it").

HTH,
cheers
Anders
--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ftdi_read loses bytes (when latency is low)

Hendrik
Hello Anders,

 > when you use the standard driver 'ftdi_sio', 'udev' will
automatically create symlinks below /dev/serial/;

I did not know that, thanks! I see now and indeed the serial number and
description of the device is there. That may help and maybe I'll make
the switch to the FTDI_SIO driver afterall.

Regards,
Hendrik



--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]  

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ftdi_read loses bytes (when latency is low)

Hendrik
In reply to this post by Uwe Bonnes
Hello Uwe,

 > So this is clearly an overrun (OE). Either handle that overrun in
your code and firmware,

Yes it is but it happens very frequently with a low latency and almost
never (I have now ran into the problem a couple of times with the
latency set to a high value too, but it is very infrequent) happens.

 > or use an asynchronous transfer, where libusb cares for you to poll
the FTDI in time,

Using async transfer did not help. Maybe the code is wrong:

tc = ftdi_read_data_submit(serialConnection->ftdi, &buffer[*bytesRead],
bytesToRead - *bytesRead);
printf("IN DONE\n");
res = ftdi_transfer_data_done (tc);
printf("AFTER DONE\n");

When the packet comes in ok, I get to 'AFTER DONE'. But when the
incomplete packet arrives it hangs in ftdi_transfer_data_done (tc)
because the number of requested bytes never come (they are lost). Does
that make sense?

 > or use the kernel driver, where the kernel is in charge to poll for
you. Hope this helps

I will look into the kernel driver or maybe I will change my software on
the microcontroller never to send packets larger than the FTDI buffer at
once. Still I am puzzled about the 'why' and 'when'. Why is this almost
never happening with a high latency and very frequently with a low
latency. Some interaction between the latency timer and libusb/libftdi
must be present because the buffer of the FTDI is never large enough to
carry the full packet.

Regards,
Hendrik

--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]  

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ftdi_read loses bytes (when latency is low)

Uwe Bonnes
In reply to this post by Hendrik
>>>>> "Hendrik" == Hendrik  <[hidden email]> writes:

...

    Hendrik> They are there but a bit hidden:
    Hendrik> http://pastebin.com/raw.php?i=3Vwk2NAs You can see that the
    Hendrik> packet with size 327 comes in with 0162 and the following
    Hendrik> packets show 0160 again.

  5.977422 1<-- 33: 0160 0202 0619 001f 4000 0100 0102 ...
  5.978486 1<-- 52: 0160 1617 1819 1a1b 1c1d 1e1f 2021 ...
  5.985341 1<-- 327: 0162 4849 4a4b 4c4d 4e4f 5051 5253 ...
  5.986459 1<-- 52: 0160 a5a6 a7a8 a9aa abac adae afb0 ...
  5.987462 1<-- 44: 0160 d7d8 d9da dbdc ddde dfe0 e1e2 ...

More remarks:
- About 8 ms elapsed between the second and the third call. This is doomed for
overflow.
- Transfers only happen in milisecond frame. Is your USB hub not USB high
speed?
- You transfer short packages due to the 1 ms latency. 1 ms with about 50
bytes is too small to fill up the USB buffer with 62 bytes.

Try to set latency to 2 ms. Then you transfer full 62 byte buffers.

Bye
--
Uwe Bonnes                [hidden email]

Institut fuer Kernphysik  Schlossgartenstrasse 9  64289 Darmstadt
--------- Tel. 06151 162516 -------- Fax. 06151 164321 ----------

--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]  

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ftdi_read loses bytes (when latency is low)

Anders Larsen
In reply to this post by Hendrik
Hello Hendrik,

On 2014-12-02 15:53, Hendrik wrote:
> Yes it is but it happens very frequently with a low latency and  
> almost never (I have now ran into the problem a couple of times with  
> the latency set to a high value too, but it is very infrequent)  
> happens.

is latency an issue for you (could you live with 16ms, where your  
problem seems not to manifest itself)?

Bear in mind that a low latency value will consume more CPU power on  
Linux, too!

Cheers
Anders
--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ftdi_read loses bytes (when latency is low)

Hendrik
In reply to this post by Uwe Bonnes
On 12/02/2014 04:04 PM, Uwe Bonnes wrote:

>      Hendrik> They are there but a bit hidden:
>      Hendrik> http://pastebin.com/raw.php?i=3Vwk2NAs You can see that the
>      Hendrik> packet with size 327 comes in with 0162 and the following
>      Hendrik> packets show 0160 again.
>
>    5.977422 1<-- 33: 0160 0202 0619 001f 4000 0100 0102 ...
>    5.978486 1<-- 52: 0160 1617 1819 1a1b 1c1d 1e1f 2021 ...
>    5.985341 1<-- 327: 0162 4849 4a4b 4c4d 4e4f 5051 5253 ...
>    5.986459 1<-- 52: 0160 a5a6 a7a8 a9aa abac adae afb0 ...
>    5.987462 1<-- 44: 0160 d7d8 d9da dbdc ddde dfe0 e1e2 ...
>
> More remarks:
> - About 8 ms elapsed between the second and the third call. This is doomed for
> overflow.
Are you sure that this is what happens or that this only seems to
happen? The 327 bytes packet consists of multiple 64 byte packets
containing the status bytes and data. I'm not completely sure if that is
what usbdump tells me, but maybe it is only the return time of that
complete transfer. So you indeed observe multiple shorted sized packets
coming in every millisecond, but the 327 bytes data transfer internally
contains 5+ 64 byte packets (though that are listed as one), so it takes
~7 ms but it also has about 6 'packets' inside so it is not strange that
it takes multiple milliseconds. With 50 bytes per millisecond being sent
to the FTDI by the microcontroller this transfer containing multiple 64
byte packets should not have caused an overflow because the transfer was
already in progress?

Still the overflow bit is set so the overflow case stands. But on the
other hand: transfers with a larger latency timer are listed as one
transfer of 522 bytes which is way over the FTDI buffer anyway, but
those cases mostly don't cause problems.

> - Transfers only happen in milisecond frame. Is your USB hub not USB high
> speed?
It is, but maybe it is not a multi-tt HUB. From what I read a full speed
device on a non multi-tt hub will force the downstream to be full speed
instead of high speed. I tried directly on my PC but that did not change
it though. Same problem.

> - You transfer short packages due to the 1 ms latency. 1 ms with about 50
> bytes is too small to fill up the USB buffer with 62 bytes.
>
> Try to set latency to 2 ms. Then you transfer full 62 byte buffers.
True but the problem persists. With the latency timer set to a high
value it 'almost never' goes wrong. I don't really understand why the
latency timer has so much influence on causing an overflow inside the FTDI.

Some percentages:

Latency vs percentage of failed transfers (in which I don't receive the
522 bytes packet)

1 -> 1.37%
2 -> 0.41%
4 -> 0.14%
16 -> 0.11%

Regards,
Hendrik




--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]  

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ftdi_read loses bytes (when latency is low)

Hendrik
In reply to this post by Anders Larsen
On 12/02/2014 04:29 PM, Anders Larsen wrote:
>> Yes it is but it happens very frequently with a low latency and
>> almost never (I have now ran into the problem a couple of times with
>> the latency set to a high value too, but it is very infrequent) happens.
>
> is latency an issue for you (could you live with 16ms, where your
> problem seems not to manifest itself)?
Yes it is. Because the protocol is 'request -> reply' it is better to
have a faster turn-around time so we can handle (a lot) more data per
second.

> Bear in mind that a low latency value will consume more CPU power on
> Linux, too!
I have seen that but the overhead seems to be far lower than on Windows.

I found out that the FTDI_SIO driver has some entries in /sys in which
you can set the latency as well so we might still go that way if this
doesn't work out. But I liked the feeling of having a more direct
connection to the FTDI like I used to on Windows. It's also quite
straight-forward what happens in case USB errors occur (ESD spikes, USB
problems, disconnections) as these are handled in the LIBUSB return
codes that are passed through by libftdi. I don't know what FTDI_SIO
does with these things as being a kernel serial port, these errors
should not even be possible (you usually have no USB problems with a
real serial port :)).

A large portion of the Windows code was about cleaning up and restarting
USB devices in case the FTDI device hung so the low-levelness was
another reason for immediately going for libftdi when switching to Linux.

Regards,
Hendrik

--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]  

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ftdi_read loses bytes (when latency is low)

Rogier Wolff
In reply to this post by Anders Larsen
On Tue, Dec 02, 2014 at 04:29:47PM +0100, Anders Larsen wrote:

> Hello Hendrik,
>
> On 2014-12-02 15:53, Hendrik wrote:
> >Yes it is but it happens very frequently with a low latency and
> >almost never (I have now ran into the problem a couple of times
> >with the latency set to a high value too, but it is very
> >infrequent) happens.
>
> is latency an issue for you (could you live with 16ms, where your
> problem seems not to manifest itself)?
>
> Bear in mind that a low latency value will consume more CPU power on
> Linux, too!

Whoa! As far as I can see we're talking about the latency timer in the
FTDI chip. The timer expires the configured time after the last
transmission to the PC. When that happens, the buffer will be sent,
full-or-not.

http://www.ftdichip.com/Support/Documents/AppNotes/AN232B-04_DataLatencyFlow.pdf

3.1, item 4.

Now read section 2.3...

The PC will assume "more data available" when the packet is 64 bytes!

So if you put the latency timer on "1ms", you'll only get one 50-byte
packet to the PC every ms. ANY delay or missed timeslot will
accumulate. There is never a chance to send more than 50 bytes in any
1ms period, because by sending less than 64 bytes the FTDI chip has
signalled "no, I don't have more data than this available at this
time."

So... For setting the latency timer: NEVER EVER set it below the time
required to gather 64 bytes. (unless your data transfer is slow
compared to 64 bytes/ms.).

        Roger.

--
** [hidden email] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
The plan was simple, like my brother-in-law Phil. But unlike
Phil, this plan just might work.

--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]  

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ftdi_read loses bytes (when latency is low)

Anders Larsen
On 2014-12-07 10:51, Rogier Wolff wrote:
> Whoa! As far as I can see we're talking about the latency timer in the
> FTDI chip.

yes indeed

> http://www.ftdichip.com/Support/Documents/AppNotes/AN232B-04_DataLatencyFlow.pdf
>
> The PC will assume "more data available" when the packet is 64 bytes!
>
> So... For setting the latency timer: NEVER EVER set it below the time
> required to gather 64 bytes. (unless your data transfer is slow
> compared to 64 bytes/ms.).

interesting info - thanks for the (important) hint!

Cheers
Anders
--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]

Loading...