ftdi.write_data doesn’t work with some python3 unicode strings

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

ftdi.write_data doesn’t work with some python3 unicode strings

Chris Brackert

Regardless of using python2 or python3, libftdi’s write_data takes a “string”. In python2 this means bytes and in python3 this means unicode.


My problem is that certain strings are not being written properly with python3. For example, when ‘\xAB’ is encoded to bytes with UTF-8 (as it does in libftdi), it turns into b’\xC2\xAB’.


I have a temporary fix that is working for me and it involves modifying the swig built file `build/python/ftdi1PYTHON_wrap.c`. In the function SWIG_AsCharPtrAndSize I replace PyUnicode_AsUTF8String with PyUnicode_AsLatin1String in line 3161. Now python3 unicode strings are converted using Latin-1 so my example ‘\xAB’ converts to the byte b’\AB’ instead of the bytes b’\xC2\xAB’.


I know this fix is not the proper way to do this but it is working for me. Can you please investigate this issue and fix the library to work with all python3 strings?


Another option I would be happy with is for ftdi.write_data to accept python3 bytes instead of unicode. As of now, if you try and actually pass the bytes b’\xAB’ to write_data, libftdi complains about the type not being correct.



libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ftdi.write_data doesn’t work with some python3 unicode strings

Thomas Jarosch
Hi Chris,

On Monday, 5. October 2015 12:18:50 Chris Brackert wrote:

> Regardless of using python2 or python3, libftdi’s write_data takes a
> “string”. In python2 this means bytes and in python3 this means unicode.
>
>
> My problem is that certain strings are not being written properly with
> python3. For example, when ‘\xAB’ is encoded to bytes with UTF-8 (as it
> does in libftdi), it turns into b’\xC2\xAB’.
>
>
> I have a temporary fix that is working for me and it involves modifying
> the swig built file `build/python/ftdi1PYTHON_wrap.c`. In the function
> SWIG_AsCharPtrAndSize I replace PyUnicode_AsUTF8String with
> PyUnicode_AsLatin1String in line 3161. Now python3 unicode strings are
> converted using Latin-1 so my example ‘\xAB’ converts to the byte b’\AB’
> instead of the bytes b’\xC2\xAB’.
>
>
> I know this fix is not the proper way to do this but it is working for me.
> Can you please investigate this issue and fix the library to work with
> all python3 strings?
>
>
> Another option I would be happy with is for ftdi.write_data to accept
> python3 bytes instead of unicode. As of now, if you try and actually pass
> the bytes b’\xAB’ to write_data, libftdi complains about the type not
> being correct.

this is probably something in the swig layer that generates
the wrapper code. I don't know swig myself, but I guess we need
to configure it to do better type mapping or so.

@xantares09: Any idea about this?

Cheers,
Thomas


--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: ftdi.write_data doesn’t work with some python3 unicode strings

Michel Zou



From: [hidden email]
To: [hidden email]; [hidden email]
CC: [hidden email]
Subject: RE: ftdi.write_data doesn’t work with some python3 unicode strings
Date: Sat, 17 Oct 2015 10:16:23 +0000



> From: [hidden email]

> To: [hidden email]
> CC: [hidden email]; [hidden email]
> Subject: Re: ftdi.write_data doesn’t work with some python3 unicode strings
> Date: Fri, 16 Oct 2015 13:40:41 +0200
>
> Hi Chris,
>
> On Monday, 5. October 2015 12:18:50 Chris Brackert wrote:
> > Regardless of using python2 or python3, libftdi’s write_data takes a
> > “string”. In python2 this means bytes and in python3 this means unicode.
> >
> >
> > My problem is that certain strings are not being written properly with
> > python3. For example, when ‘\xAB’ is encoded to bytes with UTF-8 (as it
> > does in libftdi), it turns into b’\xC2\xAB’.
> >
> >
> > I have a temporary fix that is working for me and it involves modifying
> > the swig built file `build/python/ftdi1PYTHON_wrap.c`. In the function
> > SWIG_AsCharPtrAndSize I replace PyUnicode_AsUTF8String with
> > PyUnicode_AsLatin1String in line 3161. Now python3 unicode strings are
> > converted using Latin-1 so my example ‘\xAB’ converts to the byte b’\AB’
> > instead of the bytes b’\xC2\xAB’.
> >
> >
> > I know this fix is not the proper way to do this but it is working for me.
> > Can you please investigate this issue and fix the library to work with
> > all python3 strings?
> >
> >
> > Another option I would be happy with is for ftdi.write_data to accept
> > python3 bytes instead of unicode. As of now, if you try and actually pass
> > the bytes b’\xAB’ to write_data, libftdi complains about the type not
> > being correct.
>
> this is probably something in the swig layer that generates
> the wrapper code. I don't know swig myself, but I guess we need
> to configure it to do better type mapping or so.
>
> @xantares09: Any idea about this?
>
> Cheers,
> Thomas

Hi,

I added a bytes conversion fonction for write_data.
Could someone test the following patch ?

xan.



libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]



libftdi-wirte_data.patch (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Re: ftdi.write_data doesn’t work with some python3 unicode strings

Thomas Jarosch
In reply to this post by Thomas Jarosch
Hi Chris,

On Friday, 16. October 2015 13:40:41 Thomas Jarosch wrote:
> > I know this fix is not the proper way to do this but it is working for
> > me. Can you please investigate this issue and fix the library to work
> > with all python3 strings?

please test the fix Michel provided and let me know if it works for you.

Thanks,
Thomas


--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]  

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Re: ftdi.write_data doesn’t work with some python3 unicode strings

Chris Brackert
I need to get my test hardware setup again. I should be able to get to this early next week and then I will report back. Thanks!


On Fri, Oct 23, 2015 at 4:22 AM, Thomas Jarosch <[hidden email]> wrote:

Hi Chris,

On Friday, 16. October 2015 13:40:41 Thomas Jarosch wrote:
> > I know this fix is not the proper way to do this but it is working for
> > me. Can you please investigate this issue and fix the library to work
> > with all python3 strings?

please test the fix Michel provided and let me know if it works for you.

Thanks,
Thomas


--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]




libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Re: ftdi.write_data doesn’t work with some python3 unicode strings

Chris Brackert
The patch is working great. Thanks guys! The dumb workaround that I was using meant calling the library with python 3 unicode strings, but once I updated those to bytes everything worked.

The code I’m working with used to call ftdi.write_data with a third parameter for “length” but now that is not allowed. Was that unnecesarry to begin with?


On Fri, Oct 23, 2015 at 5:16 PM, Chris Brackert <[hidden email]> wrote:

I need to get my test hardware setup again. I should be able to get to this early next week and then I will report back. Thanks!


On Fri, Oct 23, 2015 at 4:22 AM, Thomas Jarosch <[hidden email]> wrote:

Hi Chris,

On Friday, 16. October 2015 13:40:41 Thomas Jarosch wrote:
> > I know this fix is not the proper way to do this but it is working for
> > me. Can you please investigate this issue and fix the library to work
> > with all python3 strings?

please test the fix Michel provided and let me know if it works for you.

Thanks,
Thomas


--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]





libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: ftdi.write_data doesn’t work with some python3 unicode strings

Michel Zou
Hi,
Glad it worked, could you test with python 2 ?
The method was not prepared for Python at all, and that length argument was not deduced from the object passed.
M.


Date: Mon, 26 Oct 2015 16:45:52 -0700
From: [hidden email]
To: [hidden email]
CC: [hidden email]
Subject: Re: Re: ftdi.write_data doesn’t work with some python3 unicode strings

The patch is working great. Thanks guys! The dumb workaround that I was using meant calling the library with python 3 unicode strings, but once I updated those to bytes everything worked.

The code I’m working with used to call ftdi.write_data with a third parameter for “length” but now that is not allowed. Was that unnecesarry to begin with?


On Fri, Oct 23, 2015 at 5:16 PM, Chris Brackert <[hidden email]> wrote:

I need to get my test hardware setup again. I should be able to get to this early next week and then I will report back. Thanks!


On Fri, Oct 23, 2015 at 4:22 AM, Thomas Jarosch <[hidden email]> wrote:

Hi Chris,

On Friday, 16. October 2015 13:40:41 Thomas Jarosch wrote:
> > I know this fix is not the proper way to do this but it is working for
> > me. Can you please investigate this issue and fix the library to work
> > with all python3 strings?

please test the fix Michel provided and let me know if it works for you.

Thanks,
Thomas


--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]






libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]



libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: ftdi.write_data doesn’t work with some python3 unicode strings

Michel Zou
Hi,
Thomas, I think you can apply this patch even if Chris doesn't answer.
M.


From: [hidden email]
To: [hidden email]
Subject: RE: ftdi.write_data doesn’t work with some python3 unicode strings
Date: Tue, 27 Oct 2015 10:33:34 +0000

Hi,
Glad it worked, could you test with python 2 ?
The method was not prepared for Python at all, and that length argument was not deduced from the object passed.
M.


Date: Mon, 26 Oct 2015 16:45:52 -0700
From: [hidden email]
To: [hidden email]
CC: [hidden email]
Subject: Re: Re: ftdi.write_data doesn’t work with some python3 unicode strings

The patch is working great. Thanks guys! The dumb workaround that I was using meant calling the library with python 3 unicode strings, but once I updated those to bytes everything worked.

The code I’m working with used to call ftdi.write_data with a third parameter for “length” but now that is not allowed. Was that unnecesarry to begin with?


On Fri, Oct 23, 2015 at 5:16 PM, Chris Brackert <[hidden email]> wrote:

I need to get my test hardware setup again. I should be able to get to this early next week and then I will report back. Thanks!


On Fri, Oct 23, 2015 at 4:22 AM, Thomas Jarosch <[hidden email]> wrote:

Hi Chris,

On Friday, 16. October 2015 13:40:41 Thomas Jarosch wrote:
> > I know this fix is not the proper way to do this but it is working for
> > me. Can you please investigate this issue and fix the library to work
> > with all python3 strings?

please test the fix Michel provided and let me know if it works for you.

Thanks,
Thomas


--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]






libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]



libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]



libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: ftdi.write_data doesn’t work with some python3 unicode strings

Chris Brackert
I apologize that I haven't been able to test with Python 2. I'm on vacation and without a computer until Monday. I will check then.



On Fri, Oct 30, 2015 at 3:57 AM, Michel Zou <[hidden email]> wrote:

Hi,
Thomas, I think you can apply this patch even if Chris doesn't answer.
M.


From: [hidden email]
To: [hidden email]
Subject: RE: ftdi.write_data doesn’t work with some python3 unicode strings
Date: Tue, 27 Oct 2015 10:33:34 +0000

Hi,
Glad it worked, could you test with python 2 ?
The method was not prepared for Python at all, and that length argument was not deduced from the object passed.
M.


Date: Mon, 26 Oct 2015 16:45:52 -0700
From: [hidden email]
To: [hidden email]
CC: [hidden email]
Subject: Re: Re: ftdi.write_data doesn’t work with some python3 unicode strings

The patch is working great. Thanks guys! The dumb workaround that I was using meant calling the library with python 3 unicode strings, but once I updated those to bytes everything worked.

The code I’m working with used to call ftdi.write_data with a third parameter for “length” but now that is not allowed. Was that unnecesarry to begin with?


On Fri, Oct 23, 2015 at 5:16 PM, Chris Brackert <[hidden email]> wrote:

I need to get my test hardware setup again. I should be able to get to this early next week and then I will report back. Thanks!


On Fri, Oct 23, 2015 at 4:22 AM, Thomas Jarosch <[hidden email]> wrote:

Hi Chris,

On Friday, 16. October 2015 13:40:41 Thomas Jarosch wrote:
> > I know this fix is not the proper way to do this but it is working for
> > me. Can you please investigate this issue and fix the library to work
> > with all python3 strings?

please test the fix Michel provided and let me know if it works for you.

Thanks,
Thomas


--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]






libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]



libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]



libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]





libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: ftdi.write_data doesn’t work with some python3 unicode strings

Chris Brackert
I have confirmed that the patch is also working with Python 2 for my code. Thanks!


On Fri, Oct 30, 2015 at 7:59 AM, Chris Brackert <[hidden email]> wrote:

I apologize that I haven't been able to test with Python 2. I'm on vacation and without a computer until Monday. I will check then.



On Fri, Oct 30, 2015 at 3:57 AM, Michel Zou <[hidden email]> wrote:

Hi,
Thomas, I think you can apply this patch even if Chris doesn't answer.
M.


From: [hidden email]
To: [hidden email]
Subject: RE: ftdi.write_data doesn’t work with some python3 unicode strings
Date: Tue, 27 Oct 2015 10:33:34 +0000

Hi,
Glad it worked, could you test with python 2 ?
The method was not prepared for Python at all, and that length argument was not deduced from the object passed.
M.


Date: Mon, 26 Oct 2015 16:45:52 -0700
From: [hidden email]
To: [hidden email]
CC: [hidden email]
Subject: Re: Re: ftdi.write_data doesn’t work with some python3 unicode strings

The patch is working great. Thanks guys! The dumb workaround that I was using meant calling the library with python 3 unicode strings, but once I updated those to bytes everything worked.

The code I’m working with used to call ftdi.write_data with a third parameter for “length” but now that is not allowed. Was that unnecesarry to begin with?


On Fri, Oct 23, 2015 at 5:16 PM, Chris Brackert <[hidden email]> wrote:

I need to get my test hardware setup again. I should be able to get to this early next week and then I will report back. Thanks!


On Fri, Oct 23, 2015 at 4:22 AM, Thomas Jarosch <[hidden email]> wrote:

Hi Chris,

On Friday, 16. October 2015 13:40:41 Thomas Jarosch wrote:
> > I know this fix is not the proper way to do this but it is working for
> > me. Can you please investigate this issue and fix the library to work
> > with all python3 strings?

please test the fix Michel provided and let me know if it works for you.

Thanks,
Thomas


--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]






libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]



libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]



libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]






libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RE: ftdi.write_data doesn’t work with some python3 unicode strings

Thomas Jarosch
On Wednesday, 4. November 2015 14:51:36 Chris Brackert wrote:
> I have confirmed that the patch is also working with Python 2 for my code.
> Thanks!

thanks for testing, Chris!

I do get a compiler warning though with the new code:

libftdi/build/python/ftdi1PYTHON_wrap.c: In function ‘_wrap_write_data’:
libftdi/build/python/ftdi1PYTHON_wrap.c:3618:47: warning: passing argument 2 of ‘str2charp_size’ from incompatible pointer type [-Wincompatible-pointer-types]
   arg2 = (unsigned char*)str2charp_size(obj1, &arg3);
                                               ^
libftdi/build/python/ftdi1PYTHON_wrap.c:3010:15: note: expected ‘long int *’ but argument is of type ‘int *’
 inline char * str2charp_size(PyObject* pyObj, long * size)

@xantares09: Should I just change the type to "int *size"?

Thomas


--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: ftdi.write_data doesn’t work with some python3 unicode strings

Michel Zou
Ok,

We'll have to cast to (Py_ssize_t*) then:

inline char * str2charp_size(PyObject* pyObj, int * size)
{
  char * v_ = 0;
#if PY_MAJOR_VERSION >= 3
  PyBytes_AsStringAndSize(pyObj, &v_, (Py_ssize_t*)size);
#else
  PyString_AsStringAndSize(pyObj, &v_, (Py_ssize_t*)size);
#endif
  return v_;
}


> From: [hidden email]

> To: [hidden email]
> CC: [hidden email]; [hidden email]
> Subject: Re: RE: ftdi.write_data doesn’t work with some python3 unicode strings
> Date: Thu, 5 Nov 2015 13:03:23 +0100
>
> On Wednesday, 4. November 2015 14:51:36 Chris Brackert wrote:
> > I have confirmed that the patch is also working with Python 2 for my code.
> > Thanks!
>
> thanks for testing, Chris!
>
> I do get a compiler warning though with the new code:
>
> libftdi/build/python/ftdi1PYTHON_wrap.c: In function ‘_wrap_write_data’:
> libftdi/build/python/ftdi1PYTHON_wrap.c:3618:47: warning: passing argument 2 of ‘str2charp_size’ from incompatible pointer type [-Wincompatible-pointer-types]
> arg2 = (unsigned char*)str2charp_size(obj1, &arg3);
> ^
> libftdi/build/python/ftdi1PYTHON_wrap.c:3010:15: note: expected ‘long int *’ but argument is of type ‘int *’
> inline char * str2charp_size(PyObject* pyObj, long * size)
>
> @xantares09: Should I just change the type to "int *size"?
>
> Thomas
>
>
> --
> libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
> To unsubscribe send a mail to [hidden email]
>


libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RE: ftdi.write_data doesn’t work with some python3 unicode strings

Thomas Jarosch
On Thursday, 5. November 2015 15:13:52 Michel Zou wrote:

> inline char * str2charp_size(PyObject* pyObj, int * size)
> {
>   char * v_ = 0;
> #if PY_MAJOR_VERSION >= 3
>   PyBytes_AsStringAndSize(pyObj, &v_, (Py_ssize_t*)size);
> #else
>   PyString_AsStringAndSize(pyObj, &v_, (Py_ssize_t*)size);
> #endif
>   return v_;
> }

thanks, looking good! Patch committed with the above change.

Cheers,
Thomas


--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]  

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RE: ftdi.write_data doesn’t work with some python3 unicode strings

Chris Brackert
After closer inspection, I can confirm that I see that warning too. It did not appear to affect functionality.


On Thu, Nov 5, 2015 at 8:57 AM, Thomas Jarosch <[hidden email]> wrote:

On Thursday, 5. November 2015 15:13:52 Michel Zou wrote:


> inline char * str2charp_size(PyObject* pyObj, int * size)
> {
> char * v_ = 0;
> #if PY_MAJOR_VERSION >= 3
> PyBytes_AsStringAndSize(pyObj, &v_, (Py_ssize_t*)size);
> #else
> PyString_AsStringAndSize(pyObj, &v_, (Py_ssize_t*)size);
> #endif
> return v_;
> }

thanks, looking good! Patch committed with the above change.

Cheers,
Thomas


--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]




libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to [hidden email]


Loading...